Questions about Self Monitoring Systems blog post

Elastic Stack Beats
12

@jerrac / @Jerry_Hoffmeister ,

I have played with this a little bit more, and have come up with the following mash up. The cool thing about this set up is that you end up with all your Hosts and Services in the Nagios dashboard , looking very tidy, with zero nagios config. Additionally, we are sending all statuses through, including OK statuses, so service will recover properly.

Configure Logstash to Output Nagios Beats to NSCA

The only caveat about this configuration (which is not really a huge deal) is that you have to duplicate the output for each status. I am working with the logstash folks to determine why it isn't possible to use an integer value for the status code setting

This configuration takes each nagios check, and outputs it using the NSCA (Nagios Service Check Adapter).

input {
 beats {
  port => 5044
 }
}

output{
  if [status_code] == 0  {
    nagios_nsca {
      host => "localhost"
      nagios_service => "%{name}"
      nagios_status => 0
      nagios_host => "%{[beat][hostname]}"
      message_format => "%{name}: %{message}"
    }
  }
  if [status_code] == 1  {
    nagios_nsca {
      host => "localhost"
      nagios_service => "%{name}"
      nagios_status => 1
      nagios_host => "%{[beat][hostname]}"
      message_format => "%{name}: %{message}"
    }
  }
  if [status_code] == 2  {
    nagios_nsca {
      host => "localhost"
      nagios_service => "%{name}"
      nagios_status => 2
      nagios_host => "%{[beat][hostname]}"
      message_format => "%{name}: %{message}"
    }
  }
  if [status_code] == 3  {
    nagios_nsca {
      host => "localhost"
      nagios_service => "%{name}"
      nagios_status => 3
      nagios_host => "%{[beat][hostname]}"
      message_format => "%{name}: %{message}"
    }
  }
}

To make this work, you also have to install nsca-client on the Logstash server.

yum install nsca-client

Luckily for me, it put the binary in the default location of [the output setting] (https://www.elastic.co/guide/en/logstash/current/plugins-outputs-nagios_nsca.html#plugins-outputs-nagios_nsca-send_nsca_bin) - send_nsca_bin => "/usr/sbin/send_nsca"

Configure NSCA & Radar on the Nagios server

On my CentOS box, I only had to do a:

yum install nsca

I used Radar to scan for new Hosts and Services and automatically update the Nagios configuration. It's a simple script you can download and run in cron.

Here are the steps I performed to integrate Radar:

  1. I had to add a Perl dependency with yum install perl-File-Pid
  2. I set the Radar script to run in cron every so often, followed by a service nagios reload
  3. I created the file /etc/nagios/objects/radar.cfg and installed the service and host templates as defined in the Radar docs. Note that the Radar templates include reference to a host and service group which doesnt exist, so you should just remove those lines if you don't need the groups. Reference that file in /etc/nagios/nagios.cfg so that it loads.
  4. Modify /etc/nagios/nagios.cfg to enable the /etc/nagios/conf.d/ directory, as it's not enabled by default.
  5. Modify the Radar script configuration to match your environments. In my CentOS environment, I did it like this:
my $NAGIOS_LOGFILE="/var/log/nagios/nagios.log";
my $CFG_DIRECTORY="/etc/nagios/conf.d/";
my $NAGIOS_CONFIG="/etc/nagios/objects/";
my $HOST_TEMPLATES="generic-radar-host";
my $SERVICE_TEMPLATES="generic-radar-service";
my $ICINGA_USER="nagios";
my $ICINGA_GROUP="nagios";
my $ENABLE_LOGGING=1;
my $LOGFILE_DIRECTORY="/var/log/nagios/";
my $PID_FILE_DIRECTORY="/var/run/";

I was surprised at how easy this was to set up, and it found a nice range of hosts and services generated by my Beats test machines, and it looked great in nagios! You can always edit the .cfg files created by Radar and remove things that no longer exist anymore.

I haven't tried this yet, but it should also be possible to play with the host templates in order to automatically set up pings to discovered hosts as well.