Configure Logstash to Output Nagios Beats to NSCA - Nagios logs?

[safehouse]

In this post Questions about Self Monitoring Systems blog post there is mention of pushing data to Nagios using beats, which is very interesting.

I have tested the checknagiosbeat plugin and it's great. However, I wonder if anyone knows if Nagios will still generate log files when using Nagios Core as the output for the nagioscheckbeat plugin?

I ask because I use Nagios logs to collect metrics on the data being monitored. So, I either still need to collect Nagios logs or find a way to use Nagioscheckbeat to store the metrics (instead of just the status).

@PhaedrusTheGreek

[PhaedrusTheGreek (Jay Greenberg)]

Thanks for the question,

nagioscheckbeat generates 2 types of messages:

1. Check Results
2. Performance Metrics

(1) is output to Nagios Core for monitoring functionality
(2) could be forwarded directly to Elasticsearch for Visualization

In that case, you could do something like this (theoretically, untested):

output {

  if [type] == "nagiosmetric" {
    elasticsearch { 
       ... 
    }
  } else {
    nagios_nsca {
      ...
    }
  }

}

Let us know if that works!

[safehouse]

@PhaedrusTheGreek thanks for the guidance on this. I didn't realize (I should have) that nagioscheckbeat outputs both metrics and status in different types. This is great!

I've set up a test where one server is running nagioscheckbeat and outputting that data to my logstash server. I've configured a new pipeline to accept the "nagiosmetric" data and that is working well

I'm having a problem getting nagios_nsca to function. I keep getting

error=>#<TypeError: no implicit conversion of Integer into String

Here is my config:

input {
beats {
	port => 5074 
            type => "nagioscheckbeats"
	ssl => true
	ssl_certificate_authorities => ["/etc/logstash/certs/cacert.pem"]
	ssl_certificate => "/etc/logstash/certs/{{ inventory_hostname }}.crt"
	ssl_key => "/etc/logstash/certs/{{ inventory_hostname }}-logstash.pem"
	ssl_verify_mode => "force_peer"
  }
}

filter {
		if [type] =~ "nagiosmetric" {
			mutate {
			  convert => { "name" => "string" }
			  convert => { "status_code" => "string" }
			  convert => { "message" => "string" }
			}
		}
	}
	
output {

		if [type] == "nagiosmetric" {
		  elasticsearch { 
			hosts => [{{ logstash_elasticsearch_hosts }}]
			user => {{ xpack_management_elasticsearch_username }}
			password => {{ xpack_management_elasticsearch_password }}
			cacert => "{{ logstash_ssl_dir }}/{{ logstash_ssl_certificate_file }}"
			ssl_certificate_verification => false
			index => "nagioscheckbeat-%{+YYYY.MM.dd}" 
		}
		} else {
		    nagios_nsca {
				host => "{{nagios_server}}"
				port => 5667
				nagios_service => "%{name}"
				nagios_status => "%{[status_code]}"
				nagios_host => "%{[beat][hostname]}"
				message_format => "%{name}: %{message}"
			}
			# used for debugging
			#file {
			#  	path => "/tmp/logstash-output.txt"
			#  	codec => line { format => "custom format: %{message}"}
			#}
		}
	}

The debug output is:

[2019-08-31T02:12:26,725][DEBUG][logstash.outputs.nagiosnsca] Running send_nsca command     {:nagios_nsca_command=>"/usr/sbin/send_nsca -H xxxx.xxxx.com -p 5667 -d :", :message=>"xxxx.xxxx.com:heartbeat:0:heartbeat: OK: Hello<br/>"}

[2019-08-31T02:12:26,727][WARN ][logstash.outputs.nagiosnsca] Skipping nagios_nsca output; error calling send_nsca {:error=>#<TypeError: no implicit conversion of Integer into String>, :nagio
s_nsca_command=>"/usr/sbin/send_nsca -H xxxx.xxxx.com -p 5667 -d :", :message=>"xxxx.xxxx.com:heartbeat:0:heartbeat: OK: Hello<br/>", :missed_event=>#<LogStash::Event:0x669acbab>}   
[2019-08-31T02:12:26,727][DEBUG][logstash.outputs.nagiosnsca] Backtrace {:backtrace=>["org/jruby/RubyProcess.java:1567:in `spawn'", "org/jruby/RubyKernel.java:1646:in `spawn'", "uri:classload
er:/META-INF/jruby.home/lib/ruby/stdlib/open3.rb:202:in `popen_run'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/open3.rb:98:in `popen3'", "/usr/share/logstash/vendor/bundle/jruby/
2.5.0/gems/logstash-output-nagios_nsca-3.0.6/lib/logstash/outputs/nagios_nsca.rb:135:in `send_to_nagios'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-nagios_nsca-3.0.
6/lib/logstash/outputs/nagios_nsca.rb:110:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in `block in multi_receive'", "org/jruby/RubyArray.java:1792:in `ea
ch'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:118:in `multi_receive'", "org/logstash/co
nfig/ir/compiler/AbstractOutputDelegatorExt.java:101:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:235:in `block in start_workers'"]}

which seems to indicate that the error is on line 135 of the nagios_nsca.rb file. I'm not very familiar with Ruby, so this is my best attempt to track down the root of the error, but I accept that it's likely that I'm wrong.

line 104 of nagios_nsca.rb says:

message = "#{nagios_host}:#{nagios_service}:#{status}:#{msg}"

So, I can see how it's combining multiple variables into one - which might be where the integer is getting pulled in. But, I don't see any integer. I've even tried to use mutate to convert fields to strings, but that didn't seem to affect anything.

Line 135 of nagios_nsca.cfg says:

i.puts(message) 

Where is the integer? How do I remove this error? Any help would be greatly appreciated.

[PhaedrusTheGreek (Jay Greenberg)]

Maybe there's a bug in the plugin. I haven't tested this, but you could try editing nagios_nsca.rb and changing line 104 to this:

message = "#{nagios_host}:#{nagios_service}:#{status.to_s}:#{msg}"

I saw another similar report that was never resolved. Let me know if that fixes it, and we can dig deeper in the week.

[safehouse]

@PhaedrusTheGreek - I know next to nothing about Ruby, but I messed around with the plugin code enough to narrow the problem down to the cmd variable was trying to join strings with the port as an integer.

I modified the code enough to be able to see the messages coming through to Nagios. The below is NOT meant to represent an improvement on the plugin code - it's just a hack for me to locate the problem and prove that the message can get through. Hopefully, someone else can provide the correct solution in ruby.

I changed this:

  def cmd
    return @cmd if @cmd
   @cmd = [@send_nsca_bin, "-H", @host, "-p", @port.to_s, "-d", ":"]
   @cmd = @cmd + ["-c", @send_nsca_config]  if @send_nsca_config
   @cmd
 end

 def send_to_nagios(cmd, message)
   Open3.popen3(*cmd) do |i, o, e|
     i.puts(message) 	
     i.close
  end
end

to this:

  def cmd
    return @cmd if @cmd
   # @cmd = [@send_nsca_bin, "-H", @host, "-p", @port, "-d", ":"]
   # @cmd = @cmd + ["-c", @send_nsca_config]  if @send_nsca_config
   @cmd = @send_nsca_bin.to_s + " " + "-H" + " " + @host.to_s + " " + "-p" + " " + @port.to_s + " " + "-d" + " " + ":"
  # @cmd = @cmd + [" " + "-c" + " " + @send_nsca_config  if @send_nsca_config
  @cmd
end

def send_to_nagios(cmd, message)
  Open3.popen3(*cmd) do |i, o, e|
    # added logger for testing
    logger.info("Running send_nsca command", :nagios_nsca_command => cmd, :message => message)
    i.puts(message)
    i.close
   end
end