Github queries

Found something amiss in NEMS? Confirm first that you are running the latest version, and then post your bug report here.
gpb500
Junior Member
Posts: 7
Joined: Mon Jul 02, 2018 10:20 am

Github queries

Post by gpb500 »

Robbie, new user here, thought I'd chime in with a note I'm also getting the issue that was reported about the copy collector config failure (I assume everyone is).  I know you know about that, but I wanted to ask about another issue...I noticed the Rpi is hitting github a couple times per minute (as reported in Pi-hole).  Just wanted to confirm there's a reason for that and if that is right or not.  

Also a big thanks for putting this all together!

-- Greg
Last edited by gpb500 on Sat Jul 14, 2018 9:03 am, edited 1 time in total.
gpb500
Junior Member
Posts: 7
Joined: Mon Jul 02, 2018 10:20 am

RE: Github queries

Post by gpb500 »

Is the copy collector config failure the reason NConf fails to apply at least some updates?  As a work-around I can manually edit the nagios config file directly which seems to be the only option at the moment.  An example of one change I wanted to make but couldn't was increasing a notification interval from one hour to eight hours for a specific host.  The file edited directly was: /etc/nems/conf/Default_collector/hosts.cfg, save and restart nagios.

I'm just playing with this to gain some knowledge of linux, hobby-style.  So if you're too busy to reply, no worries.  Thanks!

-- Greg

PS - I blocked github access with no negative effects to nems running...I unblock to update and then reblock.
Last edited by gpb500 on Mon Jul 16, 2018 3:43 pm, edited 1 time in total.
User avatar
Robbie Ferguson
Posting Freak
Posts: 835
Joined: Wed Mar 07, 2012 3:23 pm
Location: Ontario, Canada
Contact:

RE: Github queries

Post by Robbie Ferguson »

Hi Greg,
I issued a patch for the Default_collector bug. Should be fixed for you after a sudo nems-quickfix

Re. Github.
"PS - I blocked github access with no negative effects to nems running...I unblock to update and then reblock."
Please do not do this. Yes, there is good reason why you're seeing Github getting touched... and the Default_collector bug patch is one :)

In order to prevent errors and even data corruption, it was important to have scripts in NEMS determine if the Internet connection was up before running certain tasks. That makes sense for such a server. However, in some geographic areas, it was found that while Internet was up, Github was inaccessible (DNS issue). So those users' NEMS servers could become corrupt when a nightly maintenance task (for example) would say "Yes, Internet is up" and then try to get the files from Github and fail.

The process is perfectly benign. Please see: https://github.com/Cat5TV/nems-scripts/ ... fo.sh#L155

The reason you see it periodically and rather often is simply because I check that the Internet is up before running tasks in the background.

Does that help clarify things?

Blocking Github will break some of the best bits of NEMS (see for example, nems-quickfix, which runs as a cron task every day). Plus, blocking Github will cause your NEMS server to think it has no connectivity, which would cause it to skip over maintenance tasks, would break off site backup (if you use that) and so-on.

Robbie
Robbie Ferguson // The Bald Nerd

Did I help you out? Appreciate what I do? Please consider saying thanks:
gpb500
Junior Member
Posts: 7
Joined: Mon Jul 02, 2018 10:20 am

RE: Github queries

Post by gpb500 »

Thanks Robbie...I removed the firewall rule, let it settle for 15 minutes or so, then ran quickfix, didn't appear fix it.  Then ran nems-update followed by nems-quickfix and no change.  Still get this message when trying to deploy (under copy collector config):

source_file
FAILED
Source file does not exist (/tmp/Default_collector/)


Please advise, when you have time.  Again, not critical on my end.  Thx!

-- Greg

PS - Thanks for adding clarity on the github queries.  :)

Edit:  How can I verify the update applied properly?
Last edited by gpb500 on Mon Jul 16, 2018 3:44 pm, edited 1 time in total.
User avatar
Robbie Ferguson
Posting Freak
Posts: 835
Joined: Wed Mar 07, 2012 3:23 pm
Location: Ontario, Canada
Contact:

RE: Github queries

Post by Robbie Ferguson »

Yeah it doesn't sound like the updates are working.

Was there any error output on nems-update?
Robbie Ferguson // The Bald Nerd

Did I help you out? Appreciate what I do? Please consider saying thanks:
gpb500
Junior Member
Posts: 7
Joined: Mon Jul 02, 2018 10:20 am

RE: Github queries

Post by gpb500 »

No errors.  As below:

sudo nems-quickfix
[sudo] password for "xxxxxxxx":
Performing NEMS QuickFix...
It's really just a fancy name: this may take a while.
Do not stop this script once it is running.
Please wait patiently... Done.
User avatar
Robbie Ferguson
Posting Freak
Posts: 835
Joined: Wed Mar 07, 2012 3:23 pm
Location: Ontario, Canada
Contact:

RE: Github queries

Post by Robbie Ferguson »

That's not what I asked. :P

As requested, can you please run: sudo nems-update
... and report back if there are errors?

Thanks!!
Robbie Ferguson // The Bald Nerd

Did I help you out? Appreciate what I do? Please consider saying thanks:
gpb500
Junior Member
Posts: 7
Joined: Mon Jul 02, 2018 10:20 am

RE: Github queries

Post by gpb500 »

Oops...sorry 'bout that.  No errors...ran nems-update and re-tested, deploy still fails.
sudo nems-update
PING google.com (172.217.9.78) 56(84) bytes of data.
64 bytes from ord38s09-in-f14.1e100.net (172.217.9.78): icmp_seq=1 ttl=53 time=19.1 ms

--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 19.111/19.111/19.111/0.000 ms
Updating nems-migrator... Already up-to-date.
Done.
Updating nems-www... Already up-to-date.
Done.
Updating nems-admin... Already up-to-date.
Done.
Updating nems-conf... Already up-to-date.
Done.
Updating nems-scripts... remote: Counting objects: 6, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 6 (delta 4), reused 4 (delta 2), pack-reused 0
Unpacking objects: 100% (6/6), done.
From https://github.com/Cat5TV/nems-scripts
   efc6ddd..ceb389e  master     -> origin/master
Updating efc6ddd..ceb389e
Fast-forward
 mailtest.sh | 32 ++++++++++++++++++++++++++++++--
 1 file changed, 30 insertions(+), 2 deletions(-)
Done.
Running updates and fixes... Please wait for apt tasks to complete...
Done.
Ign:1 http://download.webmin.com/download/repository sarge InRelease
Hit:2 http://download.webmin.com/download/repository sarge Release
Hit:3 http://raspbian.raspberrypi.org/raspbian stretch InRelease
Ign:4 http://webmin.mirror.somersettechsoluti ... repository sarge InRelease
Hit:5 http://archive.raspberrypi.org/debian stretch InRelease
Get:6 http://ftp.debian.org/debian stretch-backports InRelease [91.8 kB]
Hit:7 http://webmin.mirror.somersettechsoluti ... repository sarge Release
Ign:8 http://giteduberger.fr rpimonitor/ InRelease
Hit:10 http://giteduberger.fr rpimonitor/ Release
Ign:11 https://apt.izzysoft.de/ubuntu generic InRelease
Hit:12 https://apt.izzysoft.de/ubuntu generic Release
Fetched 91.8 kB in 3s (26.5 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
50 packages can be upgraded. Run 'apt list --upgradable' to see them.
Ign:1 http://download.webmin.com/download/repository sarge InRelease
Hit:2 http://download.webmin.com/download/repository sarge Release
Hit:3 http://raspbian.raspberrypi.org/raspbian stretch InRelease
Hit:4 http://ftp.debian.org/debian stretch-backports InRelease
Hit:5 http://archive.raspberrypi.org/debian stretch InRelease
Ign:6 http://webmin.mirror.somersettechsoluti ... repository sarge InRelease
Hit:7 http://webmin.mirror.somersettechsoluti ... repository sarge Release
Ign:8 http://giteduberger.fr rpimonitor/ InRelease
Hit:9 http://giteduberger.fr rpimonitor/ Release
Ign:11 https://apt.izzysoft.de/ubuntu generic InRelease
Hit:12 https://apt.izzysoft.de/ubuntu generic Release
Reading package lists... Done
Building dependency tree
Reading state information... Done
50 packages can be upgraded. Run 'apt list --upgradable' to see them.
Done.
User avatar
Robbie Ferguson
Posting Freak
Posts: 835
Joined: Wed Mar 07, 2012 3:23 pm
Location: Ontario, Canada
Contact:

RE: Github queries

Post by Robbie Ferguson »

Thanks. Can you do a reboot for me on the NEMS server and then try generating your config?

Also, was your server deployed as a new server, or did you use Migrator to restore an old backup?
Robbie Ferguson // The Bald Nerd

Did I help you out? Appreciate what I do? Please consider saying thanks:
gpb500
Junior Member
Posts: 7
Joined: Mon Jul 02, 2018 10:20 am

RE: Github queries

Post by gpb500 »

Rebooted, regenerated, ran deploy...but still get the same error "Source file does not exist (/tmp/Default_collector/)". I was on 1.3 for a couple weeks just as you were working on 1.4 and I think what I did was attempted to migrate but that failed or caused some issue, and in the end I just added all the hosts back manually. What I can't recall is if I recreated the image on the memory card or simply ran nems-init and started over.
Post Reply