My Raspberrys all take a break at the same time

Having issues? Post them here, and help other users.
Post Reply
User avatar
kd4pyr
Junior Member
Posts: 28
Joined: Wed Jul 05, 2017 4:56 pm

My Raspberrys all take a break at the same time

Post by kd4pyr »

I have a few Raspberry PIs that I monitor with NEMS. It seems they all take a break at 3:00 am every Sunday morning.
I know this is not a NEMS question, but is there some process that happens on the PIs at the same time each week that would cause this? I poked around scheduled jobs and nothing jumps out at me. Thanks.

Rick

Here is a list of the logged alerts:

[2020-06-21 03:49:13] SERVICE ALERT: Raspberry Pi - NEMS;NEMS Current Load;OK;HARD;4;OK - load average: 0.08, 0.40, 5.91
[2020-06-21 03:49:13] SERVICE ALERT: Raspberry Pi - PiHole;NEMS Current Load;OK;HARD;4;OK - load average: 0.08, 0.40, 5.91
[2020-06-21 03:49:13] SERVICE ALERT: Raspberry Pi - Kodi;NEMS Current Load;OK;HARD;4;OK - load average: 0.08, 0.40, 5.91
[2020-06-21 03:49:13] SERVICE ALERT: Raspberry Pi - Network Status Display;NEMS Current Load;OK;HARD;4;OK - load average: 0.08, 0.40, 5.91
[2020-06-21 03:49:13] SERVICE ALERT: Raspberry Pi - DShield;NEMS Current Load;OK;HARD;4;OK - load average: 0.08, 0.40, 5.91
[2020-06-21 03:44:13] SERVICE ALERT: Raspberry Pi - PiHole;NEMS Current Load;WARNING;HARD;4;WARNING - load average: 0.14, 0.84, 8.11
[2020-06-21 03:44:13] SERVICE ALERT: Raspberry Pi - NEMS;NEMS Current Load;WARNING;HARD;4;WARNING - load average: 0.14, 0.84, 8.11
[2020-06-21 03:44:13] SERVICE ALERT: Raspberry Pi - Kodi;NEMS Current Load;WARNING;HARD;4;WARNING - load average: 0.14, 0.84, 8.11
[2020-06-21 03:44:13] SERVICE ALERT: Raspberry Pi - Network Status Display;NEMS Current Load;WARNING;HARD;4;WARNING - load average: 0.14, 0.84, 8.11
[2020-06-21 03:44:13] SERVICE ALERT: Raspberry Pi - DShield;NEMS Current Load;WARNING;HARD;4;WARNING - load average: 0.14, 0.84, 8.11
[2020-06-21 03:26:39] SERVICE ALERT: Raspberry Pi - Network Status Display;PING;OK;SOFT;2;PING OK - Packet loss = 16%, RTA = 85.76 ms
[2020-06-21 03:25:34] SERVICE ALERT: Raspberry Pi - Network Status Display;PING;WARNING;SOFT;1;PING WARNING - Packet loss = 16%, RTA = 103.02 ms
[2020-06-21 03:24:29] SERVICE ALERT: Raspberry Pi - Network Status Display;PING;OK;SOFT;2;PING OK - Packet loss = 16%, RTA = 90.67 ms
[2020-06-21 03:24:13] SERVICE ALERT: Raspberry Pi - NEMS;NEMS Current Load;CRITICAL;HARD;4;CRITICAL - load average: 4.35, 36.41, 28.95
[2020-06-21 03:24:13] SERVICE ALERT: Raspberry Pi - PiHole;NEMS Current Load;CRITICAL;HARD;4;CRITICAL - load average: 4.35, 36.41, 28.95
[2020-06-21 03:24:13] SERVICE ALERT: Raspberry Pi - Kodi;NEMS Current Load;CRITICAL;HARD;4;CRITICAL - load average: 4.35, 36.41, 28.95
[2020-06-21 03:24:13] SERVICE ALERT: Raspberry Pi - Network Status Display;NEMS Current Load;CRITICAL;HARD;4;CRITICAL - load average: 4.35, 36.41, 28.95
[2020-06-21 03:24:13] SERVICE ALERT: Raspberry Pi - DShield;NEMS Current Load;CRITICAL;HARD;4;CRITICAL - load average: 4.35, 36.41, 28.95
[2020-06-21 03:23:23] SERVICE ALERT: Raspberry Pi - Network Status Display;PING;WARNING;SOFT;1;PING WARNING - Packet loss = 16%, RTA = 103.03 ms
[2020-06-21 03:23:13] SERVICE ALERT: Raspberry Pi - NEMS;NEMS Current Load;CRITICAL;SOFT;3;CRITICAL - load average: 11.63, 44.48, 30.88
[2020-06-21 03:23:13] SERVICE ALERT: Raspberry Pi - PiHole;NEMS Current Load;CRITICAL;SOFT;3;CRITICAL - load average: 11.63, 44.48, 30.88
[2020-06-21 03:23:13] SERVICE ALERT: Raspberry Pi - Kodi;NEMS Current Load;CRITICAL;SOFT;3;CRITICAL - load average: 11.63, 44.48, 30.88
[2020-06-21 03:23:13] SERVICE ALERT: Raspberry Pi - Network Status Display;NEMS Current Load;CRITICAL;SOFT;3;CRITICAL - load average: 11.63, 44.48, 30.88
[2020-06-21 03:23:13] SERVICE ALERT: Raspberry Pi - DShield;NEMS Current Load;CRITICAL;SOFT;3;CRITICAL - load average: 11.63, 44.48, 30.88
[2020-06-21 03:22:18] SERVICE ALERT: Raspberry Pi - Network Status Display;PING;OK;SOFT;2;PING OK - Packet loss = 16%, RTA = 86.26 ms
[2020-06-21 03:22:14] SERVICE ALERT: Raspberry Pi - NEMS;CPU Temperature;OK;SOFT;4;TEMPERATURE OK - CPU Temp: 69.832 °C
[2020-06-21 03:22:13] SERVICE ALERT: Raspberry Pi - NEMS;NEMS Current Load;CRITICAL;SOFT;2;CRITICAL - load average: 30.75, 54.24, 32.90
[2020-06-21 03:22:13] SERVICE ALERT: Raspberry Pi - PiHole;NEMS Current Load;CRITICAL;SOFT;2;CRITICAL - load average: 30.75, 54.24, 32.90
[2020-06-21 03:22:13] SERVICE ALERT: Raspberry Pi - Kodi;NEMS Current Load;CRITICAL;SOFT;2;CRITICAL - load average: 30.75, 54.24, 32.90
[2020-06-21 03:22:13] SERVICE ALERT: Raspberry Pi - Network Status Display;NEMS Current Load;CRITICAL;SOFT;2;CRITICAL - load average: 30.75, 54.24, 32.90
[2020-06-21 03:22:13] SERVICE ALERT: Raspberry Pi - DShield;NEMS Current Load;CRITICAL;SOFT;2;CRITICAL - load average: 30.75, 54.24, 32.90
[2020-06-21 03:21:13] SERVICE ALERT: Raspberry Pi - NEMS;CPU Temperature;CRITICAL;SOFT;3;TEMPERATURE CRITICAL - CPU Temp: 82.744 °C
[2020-06-21 03:21:13] SERVICE ALERT: Raspberry Pi - NEMS;NEMS Current Load;CRITICAL;SOFT;1;CRITICAL - load average: 87.44, 66.79, 35.09
[2020-06-21 03:21:13] SERVICE ALERT: Raspberry Pi - Kodi;NEMS Current Load;CRITICAL;SOFT;1;CRITICAL - load average: 87.44, 66.79, 35.09
[2020-06-21 03:21:13] SERVICE ALERT: Raspberry Pi - Network Status Display;NEMS Current Load;CRITICAL;SOFT;1;CRITICAL - load average: 87.44, 66.79, 35.09
[2020-06-21 03:21:13] SERVICE ALERT: Raspberry Pi - DShield;NEMS Current Load;CRITICAL;SOFT;1;CRITICAL - load average: 87.44, 66.79, 35.09
[2020-06-21 03:21:13] SERVICE ALERT: Raspberry Pi - PiHole;NEMS Current Load;CRITICAL;SOFT;1;CRITICAL - load average: 87.44, 66.79, 35.09
[2020-06-21 03:21:13] SERVICE ALERT: Raspberry Pi - Network Status Display;PING;WARNING;SOFT;1;PING WARNING - Packet loss = 0%, RTA = 100.33 ms
[2020-06-21 03:10:35] SERVICE ALERT: Raspberry Pi - NEMS;CPU Temperature;WARNING;SOFT;2;TEMPERATURE WARNING - CPU Temp: 79.516 °C
[2020-06-21 03:09:27] SERVICE ALERT: Raspberry Pi - NEMS;CPU Temperature;WARNING;SOFT;1;TEMPERATURE WARNING - CPU Temp: 77.902 °C
[2020-06-21 03:08:28] SERVICE ALERT: Raspberry Pi - Network Status Display;PING;OK;SOFT;2;PING OK - Packet loss = 16%, RTA = 87.94 ms
[2020-06-21 03:07:23] SERVICE ALERT: Raspberry Pi - Network Status Display;PING;WARNING;SOFT;1;PING WARNING - Packet loss = 16%, RTA = 101.98 ms
[2020-06-21 03:06:29] SERVICE ALERT: Raspberry Pi - PiHole;PING;OK;SOFT;2;PING OK - Packet loss = 0%, RTA = 0.74 ms
[2020-06-21 03:05:25] SERVICE ALERT: Raspberry Pi - PiHole;PING;CRITICAL;SOFT;1;CRITICAL - Plugin timed out after 10 seconds
JatBee
Junior Member
Posts: 42
Joined: Sat Nov 11, 2017 3:54 pm

Re: My Raspberrys all take a break at the same time

Post by JatBee »

Take very careful look at your setup, and make sure you are actually monitoring things on each host. I seem to recall a while back that I had attached some local monitors, to some remote hosts. the stats i was getting were actually all from the NEMS host, although they all appeared to be reported from various others.

I had the same general effect as you - they would all mysteriously go under high load at the same time.

Not saying that is what is happening here - but it fits the symptoms.

Hope it helps a little.

Jim
User avatar
kd4pyr
Junior Member
Posts: 28
Joined: Wed Jul 05, 2017 4:56 pm

Re: My Raspberrys all take a break at the same time

Post by kd4pyr »

That was it! Thank you. I did not notice that checks were using LOCAL.

Rick
Post Reply