# Questions about Self Monitoring Systems blog post

**URL:** <https://discuss.elastic.co/t/questions-about-self-monitoring-systems-blog-post/43542>\
**Category:** Beats\
**Created:** [March 4, 2016, 6:20pm UTC](https://discuss.elastic.co/t/questions-about-self-monitoring-systems-blog-post/43542 "2016-03-04T18:20:48Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [March 4, 2016, 6:20pm UTC](https://discuss.elastic.co/t/questions-about-self-monitoring-systems-blog-post/43542/1 "2016-03-04T18:20:48Z")

</div>

I found [A case for self monitoring systems | Elastic Blog](https://www.elastic.co/blog/a-case-for-self-monitoring-systems) very interesting. It's very close to what I really want to do here at work. The problem is that that setup relies on Watcher for sending alerts. There's no way we can afford a subscription, so Watcher is out of our reach. Are there any alternatives to Watcher I have not found?

The post does say this:

> As an alternative to Watcher, Integrating Elasticsearch with any  
> existing monitoring system would be trivial - one could simply set up a  
> check that alerts and escalates under similar conditions.

That sounds like we'd still need a monitoring system. Avoiding that need would be the whole point of the "Self Monitoring" set up, right?

Also, do you need to install nagios on every node that you are monitoring to make this work?

I'm posting this in the Beats forum because it's connected to the nagiosbeats beat. Apologies if it should have been posted elsewhere.

---

<div class="post-metadata">

**Author:** ![PhaedrusTheGreek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phaedrusthegreek/32/4884_2.png) [@PhaedrusTheGreek](https://discuss.elastic.co/u/PhaedrusTheGreek)\
**Post date:** [March 4, 2016, 6:40pm UTC](https://discuss.elastic.co/t/questions-about-self-monitoring-systems-blog-post/43542/2 "2016-03-04T18:40:38Z")

</div>

Hi @jerrac,

You do not need to install Nagios Core on end systems - only _plugins_ or _checks_. In some cases, the check might be a single Perl script, or in other cases it might be a python script plus dependencies, etc. Most Linux distributions have installable packs of Nagios plugins via repository, e.g., you can install many common plugins like this:

```auto
yum install nagios-plugins-all

```

or e.g., just the disk check plugin:

```auto
yum install nagios-plugins-disk

```

Even without watcher, systems are still "self monitoring", since checks are pushed by the systems into a central repository, instead of a central system reaching out to check the end systems.

Nagios Core does 2 things - Monitoring & Alerting. In our alternative, systems would continue to "self-monitor", but Nagios would handle the alerting piece.

The only outstanding task would be to develop a Nagios check to run out of Nagios Core, that checks Elasticsearch (using the same queries that Watcher would use). The advantage here is that Nagios (or whatever monitoring system) only reaches out to Elasticsearch using a single check, as opposed to hundreds or thousands.

Let me know what you think...

---

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [March 4, 2016, 7:20pm UTC](https://discuss.elastic.co/t/questions-about-self-monitoring-systems-blog-post/43542/3 "2016-03-04T19:20:32Z")

</div>

So, all nodes would need nagios plugins installed, got it.

If I still have to have nagios core running so that alerts work, then I have to ask myself why not just use nagios to do all the monitoring as well? For me, the point of the self monitoring aspect is to limit the number tools I have to support. If ELK/Beats can send the alerts as well as do the monitoring, then that's less work for me. I wouldn't have to learn how to configure nagios.

Anyway, I get that we'd need another service running to send alerts. I was just hoping someone had created an open source ES plugin that was similar to watcher.

All that said, here's a few thoughts on how we could expand on the overall idea.

Build a Kibana "monitoring" plugin. It would let you define "alert searches " and how the alerts would be sent. (I assume nodejs has email/sms/etc. plugins that could do that.) Ideally, there'd be a default dashboard that displays relevant info for the various alert searches. And there'd also be a tool that would let people define the beat config for a nagios check without knowing how to configure beats. They'd then send the generated config to whomever configures the beats. And now I'm wanting to find time to learn nodejs and see if this would even be feasible... 🙂

---

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [March 4, 2016, 10:43pm UTC](https://discuss.elastic.co/t/questions-about-self-monitoring-systems-blog-post/43542/4 "2016-03-04T22:43:55Z")

</div>

For future reference, logstash output plugins might work as a watcher replacement.

Using: [https://www.elastic.co/guide/en/logstash/current/plugins-outputs-email.html](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-email.html)

wouldn't something like this be possible?

```
if [field] == "error|critical" {
    email {
    <options>
    }
 }

```

Though, the docs don't indicate if the entire log message is sent, or just whatever you put in the config.

You could do something similar with [https://www.elastic.co/guide/en/logstash/current/plugins-outputs-hipchat.html](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-hipchat.html)

And [https://www.elastic.co/guide/en/logstash/current/plugins-outputs-nagios.html](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-nagios.html) could help avoid having to write searches to trigger nagios alerts.

[https://www.elastic.co/guide/en/logstash/current/plugins-outputs-sns.html](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-sns.html) might help with sms text messages, if that's needed.

---

<div class="post-metadata">

**Author:** ![PhaedrusTheGreek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phaedrusthegreek/32/4884_2.png) [@PhaedrusTheGreek](https://discuss.elastic.co/u/PhaedrusTheGreek)\
**Post date:** [March 5, 2016, 4:08pm UTC](https://discuss.elastic.co/t/questions-about-self-monitoring-systems-blog-post/43542/5 "2016-03-05T16:08:58Z")

</div>

@jerrac I think it would be possible to whip something simple up in Logstash:

```auto
input {
  http_poller {
    urls => {
      nagioschecks => {
        method => post
        body => '{
                    "query":{
                      "filtered":{
                         "query" : {
                            "bool" : {
                              "must" : [
                                    { "term" : {"_type": "nagioscheck"} },
                                    { "range" : {"@timestamp" : {"gte" : "now-30m"}} },
                                    { "term" : {"status" : "CRITICAL" } }
                              ]
                            }
                         }
                      }
                    }
                }'
        url => "http://localhost:9200/nagioscheckbeat*/_search"
        headers => {
          Accept => "application/json"
        }
      }
    }
    request_timeout => 30
    interval => 10
    codec => "json"
  }
}

output {
  if [hits][hits] != [] {
   email {
    to => "you@gmail.com"
   }
  }
}

```

Although it might be easier to pump the check results through Logstash inline, and then alert as you put it:

```auto
if [status] == "WARNING|CRITICAL" {
    email {
      to => "you@gmail.com"
    }
 }

```

The disadvantage to this approach would be a barrage of emails when anything went wrong, so you would need some way to throttle / acknowledge them.

---

<div class="post-metadata">

**Author:** ![Jerry\_Hoffmeister](https://avatars.discourse-cdn.com/v4/letter/j/90ced4/32.png) [@Jerry\_Hoffmeister](https://discuss.elastic.co/u/Jerry_Hoffmeister)\
**Post date:** [March 9, 2016, 1:01am UTC](https://discuss.elastic.co/t/questions-about-self-monitoring-systems-blog-post/43542/6 "2016-03-09T01:01:18Z")

</div>

This looks pretty interesting to me too - is there some example logstash config that I could use to start with if I just want to pull the data in from the example thru logstash instead of direct to ES? And is there a minimum logstash version I would need? Would 1.5.6 for example be sufficient or do I need the current version?

---

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [March 10, 2016, 5:00pm UTC](https://discuss.elastic.co/t/questions-about-self-monitoring-systems-blog-post/43542/7 "2016-03-10T17:00:15Z")

</div>

Well, I had the wonderful idea this morning of building a Kibana app that would do a lot of the monitoring tasks. Run saved searches on a schedule, find all the servers configured with a beat, show a dashboard with server status, use saved searches as a form of remote probe, etc. Then I ran into this: [https://github.com/elastic/kibana/issues/4704](https://github.com/elastic/kibana/issues/4704) So, no custom plugin development is possible for Kibana yet. Which confuses me since I thought that was exactly what Timelion was... (Well, it's possible if I want to frequently rewrite my plugin...)

@Jerry_Hoffmeister Could you clarify?

If I were to guess what you mean, you want to use nagios beat to send data to elasticsearch via logstash. Right? If so, you can do so easily with the beats input in logstash 2.x. That's what I do with file and top beat. The lumberjack input might work for 1.5.6... You could also output from nagiosbeat to a file, then use logstash-forwarder to watch that file and send to logstash.

---

<div class="post-metadata">

**Author:** ![Jerry\_Hoffmeister](https://avatars.discourse-cdn.com/v4/letter/j/90ced4/32.png) [@Jerry\_Hoffmeister](https://discuss.elastic.co/u/Jerry_Hoffmeister)\
**Post date:** [March 15, 2016, 12:19am UTC](https://discuss.elastic.co/t/questions-about-self-monitoring-systems-blog-post/43542/8 "2016-03-15T00:19:37Z")

</div>

Thanks, yeah, that's what I wanted to do but I ended up just upgrading everything to the latest versions and sending directly to elasticsearch. I'm just kinda figuring things out at this point - yes I could use the beats input which I'm currently using with filebeat. I guess unless I want to modify the data there's no advantage to going thru logstash?

---

<div class="post-metadata">

**Author:** ![monica](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/monica/32/3696_2.png) [@monica](https://discuss.elastic.co/u/monica)\
**Post date:** [March 18, 2016, 12:56pm UTC](https://discuss.elastic.co/t/questions-about-self-monitoring-systems-blog-post/43542/9 "2016-03-18T12:56:40Z")

</div>

Yes, that's right. For now, if you use Filebeat, it makes sense to send the data through Logstash as Filebeat doesn't do any parsing of your log lines. In the next major release, [Ingest Node](https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest.html) is available and you would be able to send the data directly to Elasticsearch.

---

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [March 21, 2016, 5:27pm UTC](https://discuss.elastic.co/t/questions-about-self-monitoring-systems-blog-post/43542/10 "2016-03-21T17:27:30Z")

</div>

Are there any compiled binaries of nagioscheckbeat out there I could use for testing things? I haven't quite got the hang of compiling go projects yet. I'm going to go work on that, but just downloading something would be easier when I'm just testing.

---

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [March 21, 2016, 6:26pm UTC](https://discuss.elastic.co/t/questions-about-self-monitoring-systems-blog-post/43542/11 "2016-03-21T18:26:58Z")

</div>

Never mind... [https://github.com/PhaedrusTheGreek/nagioscheckbeat/tree/master/build](https://github.com/PhaedrusTheGreek/nagioscheckbeat/tree/master/build)

---

<div class="post-metadata">

**Author:** ![PhaedrusTheGreek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phaedrusthegreek/32/4884_2.png) [@PhaedrusTheGreek](https://discuss.elastic.co/u/PhaedrusTheGreek)\
**Post date:** [March 28, 2016, 9:27pm UTC](https://discuss.elastic.co/t/questions-about-self-monitoring-systems-blog-post/43542/12 "2016-03-28T21:27:08Z")

</div>

@jerrac / @Jerry_Hoffmeister ,

I have played with this a little bit more, and have come up with the following mash up. The cool thing about this set up is that you end up with all your Hosts and Services in the Nagios dashboard , looking very tidy, with zero nagios config. Additionally, we are sending all statuses through, including OK statuses, so service will recover properly.

## Configure Logstash to Output Nagios Beats to NSCA

_The only caveat about this configuration (which is not really a huge deal) is that you have to duplicate the output for each status. I am working with the logstash folks to determine why it isn't possible to use an integer value for the status code setting_

This configuration takes each nagios check, and outputs it using the NSCA (Nagios Service Check Adapter).

```auto
input {
 beats {
  port => 5044
 }
}

output{
  if [status_code] == 0 {
    nagios_nsca {
      host => "localhost"
      nagios_service => "%{name}"
      nagios_status => 0
      nagios_host => "%{[beat][hostname]}"
      message_format => "%{name}: %{message}"
    }
  }
  if [status_code] == 1 {
    nagios_nsca {
      host => "localhost"
      nagios_service => "%{name}"
      nagios_status => 1
      nagios_host => "%{[beat][hostname]}"
      message_format => "%{name}: %{message}"
    }
  }
  if [status_code] == 2 {
    nagios_nsca {
      host => "localhost"
      nagios_service => "%{name}"
      nagios_status => 2
      nagios_host => "%{[beat][hostname]}"
      message_format => "%{name}: %{message}"
    }
  }
  if [status_code] == 3 {
    nagios_nsca {
      host => "localhost"
      nagios_service => "%{name}"
      nagios_status => 3
      nagios_host => "%{[beat][hostname]}"
      message_format => "%{name}: %{message}"
    }
  }
}

```

To make this work, you also have to install _nsca-client_ on the Logstash server.

```auto
yum install nsca-client

```

Luckily for me, it put the binary in the default location of [the output setting] ([https://www.elastic.co/guide/en/logstash/current/plugins-outputs-nagios\_nsca.html#plugins-outputs-nagios\_nsca-send\_nsca\_bin](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-nagios_nsca.html#plugins-outputs-nagios_nsca-send_nsca_bin)) - `send_nsca_bin => "/usr/sbin/send_nsca"`

## Configure NSCA & Radar on the Nagios server

On my CentOS box, I only had to do a:

```auto
yum install nsca

```

I used [Radar](https://exchange.nagios.org/directory/Addons/Passive-Checks/Radar--2D-add-hosts-and-services-automatically/details) to scan for new Hosts and Services and automatically update the Nagios configuration. It's a simple script you can download and run in cron.

Here are the steps I performed to integrate Radar:

1. I had to add a Perl dependency with `yum install perl-File-Pid`
2. I set the Radar script to run in cron every so often, followed by a `service nagios reload`
3. I created the file `/etc/nagios/objects/radar.cfg` and installed the service and host templates as defined in the Radar docs. Note that the Radar templates include reference to a host and service group which doesnt exist, so you should just remove those lines if you don't need the groups. Reference that file in `/etc/nagios/nagios.cfg` so that it loads.
4. Modify `/etc/nagios/nagios.cfg` to enable the `/etc/nagios/conf.d/` directory, as it's not enabled by default.
5. Modify the Radar script configuration to match your environments. In my CentOS environment, I did it like this:

```auto
my $NAGIOS_LOGFILE="/var/log/nagios/nagios.log";
my $CFG_DIRECTORY="/etc/nagios/conf.d/";
my $NAGIOS_CONFIG="/etc/nagios/objects/";
my $HOST_TEMPLATES="generic-radar-host";
my $SERVICE_TEMPLATES="generic-radar-service";
my $ICINGA_USER="nagios";
my $ICINGA_GROUP="nagios";
my $ENABLE_LOGGING=1;
my $LOGFILE_DIRECTORY="/var/log/nagios/";
my $PID_FILE_DIRECTORY="/var/run/";

```

I was surprised at how easy this was to set up, and it found a nice range of hosts and services generated by my Beats test machines, and it looked great in nagios! You can always edit the `.cfg` files created by Radar and remove things that no longer exist anymore.

I haven't tried this yet, but it should also be possible to play with the host templates in order to automatically set up pings to discovered hosts as well.

---

<div class="post-metadata">

**Author:** ![Jerry\_Hoffmeister](https://avatars.discourse-cdn.com/v4/letter/j/90ced4/32.png) [@Jerry\_Hoffmeister](https://discuss.elastic.co/u/Jerry_Hoffmeister)\
**Post date:** [March 28, 2016, 9:53pm UTC](https://discuss.elastic.co/t/questions-about-self-monitoring-systems-blog-post/43542/13 "2016-03-28T21:53:02Z")

</div>

Interesting... At this point, we're trying to JUST use ELK to monitor and eventually send alerts / tickets to ServiceNow. I liked the idea of being able to use any nagios plugin to send data thru beats to elasticsearch. We're NOT using nagios at this point although (not my decision) not sure that we shouldn't.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [March 29, 2016, 12:10pm UTC](https://discuss.elastic.co/t/questions-about-self-monitoring-systems-blog-post/43542/14 "2016-03-29T12:10:25Z")

</div>

Why all these if-statements? Wouldn't this do the trick:

```auto
nagios_nsca {
      host => "localhost"
      nagios_service => "%{name}"
      nagios_status => "%{[status_code]}"
      nagios_host => "%{[beat][hostname]}"
      message_format => "%{name}: %{message}"
    }

```

---

<div class="post-metadata">

**Author:** ![PhaedrusTheGreek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phaedrusthegreek/32/4884_2.png) [@PhaedrusTheGreek](https://discuss.elastic.co/u/PhaedrusTheGreek)\
**Post date:** [March 29, 2016, 12:22pm UTC](https://discuss.elastic.co/t/questions-about-self-monitoring-systems-blog-post/43542/15 "2016-03-29T12:22:03Z")

</div>

@steffens,

Discussing this yesterday with @suyograo -

The Nagios NSCA output plugin [does a check here](https://github.com/logstash-plugins/logstash-output-nagios_nsca/blob/master/lib/logstash/outputs/nagios_nsca.rb#L90) that will fail if the passed variable is not an integer.

We think this is a bug in Logstash that an integer value cannot be passed as a config parameter.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [March 29, 2016, 1:59pm UTC](https://discuss.elastic.co/t/questions-about-self-monitoring-systems-blog-post/43542/16 "2016-03-29T13:59:17Z")

</div>

TIL, thanks.

---

<div class="post-metadata">

**Author:** ![Jerry\_Hoffmeister](https://avatars.discourse-cdn.com/v4/letter/j/90ced4/32.png) [@Jerry\_Hoffmeister](https://discuss.elastic.co/u/Jerry_Hoffmeister)\
**Post date:** [May 16, 2016, 11:37pm UTC](https://discuss.elastic.co/t/questions-about-self-monitoring-systems-blog-post/43542/17 "2016-05-16T23:37:15Z")

</div>

With nagioscheckbeat, can I add fields to a document?

Also, I'd like to modify one of the fields and I'm assuming the best way is using logstash? I'm using nagioscheckbeat with a mongodb nagios plugin ([https://github.com/mzupan/nagios-plugin-mongodb](https://github.com/mzupan/nagios-plugin-mongodb)) and one of the fields that comes thru is "args" which looks like: "-H 10.0.0.202 -A connect -P 27018 -W 2 -C 4 -s -u username -p password" for example. I'd like to get rid of the password. I could use mutate to remove the entire field but I'd prefer to just redact the password. And the args field isn't consistent (there may be missing or other parameters).

---

<div class="post-metadata">

**Author:** ![Jerry\_Hoffmeister](https://avatars.discourse-cdn.com/v4/letter/j/90ced4/32.png) [@Jerry\_Hoffmeister](https://discuss.elastic.co/u/Jerry_Hoffmeister)\
**Post date:** [May 17, 2016, 12:06am UTC](https://discuss.elastic.co/t/questions-about-self-monitoring-systems-blog-post/43542/18 "2016-05-17T00:06:31Z")

</div>

Figured out the second part - redacting the password. Added the following to my logstash filter:

```
  gsub => [
    "args", "-p \S*", "-p redacted"
  ]
```

---

<div class="post-metadata">

**Author:** ![PhaedrusTheGreek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phaedrusthegreek/32/4884_2.png) [@PhaedrusTheGreek](https://discuss.elastic.co/u/PhaedrusTheGreek)\
**Post date:** [January 6, 2017, 2:23pm UTC](https://discuss.elastic.co/t/questions-about-self-monitoring-systems-blog-post/43542/19 "2017-01-06T14:23:14Z")

</div>

@Jerry_Hoffmeister - You can statically add fields directly to each document using the [fields directive](https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-general.html#libbeat-configuration-fields), part of libbeat.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 9:50pm UTC](https://discuss.elastic.co/t/questions-about-self-monitoring-systems-blog-post/43542/20 "2017-07-05T21:50:06Z")

</div>