Logstash with zabbix output ISSUE


(Marko) #1

Hello!

I have problem, my zabbix output just don't work and i can't find reason why.

For example, i've created simple logstash conf file:

input {
tcp {
port => 6000
type => syslog
}
udp {
port => 6000
type => syslog
}
}
filter {
grep {
type => "linux-syslog"
match => [ "@message", "(error|ERROR|CRITICAL)" ]
add_tag => [ "zabbix-sender" ]
add_field => [
"zabbix_host", "%{@test01}",
"zabbix_item", "Test.broj"
]
}
}

output {
stdout { codec => rubydebug }
zabbix {
tags => "zabbix-sender"
zabbix_sender => "/usr/local/bin/zabbix_sender"
}
}

Then i created Host in zabbix named re01os01.net.ot.hr, zabbix trapper item named "Test" with key "broj" and type of information as "TEXT".

  1. I start logstash (it started well with no errors)
  2. I telnet to localhost port 6000 and type "CRITICAL"
    Logstash return me parsed log:
    {
    "message" => "CRITICAL\r",
    "@version" => "1",
    "@timestamp" => "2015-05-28T00:08:34.282Z",
    "host" => "0:0:0:0:0:0:0:1:50210",
    "type" => "syslog"
    }

So it works great!

After that, i go to zabbix and check trapper item and it just don't get any value.

Please help!

Best Regards,

Marko


(Aaron Mildenstein) #2

What version of Logstash are you using? The configuration you're using here seems to indicate you're using Logstash 1.4.x or older. If you're using Logstash 1.5, the Zabbix output plugin is totally changed.

See https://www.elastic.co/guide/en/logstash/current/plugins-outputs-zabbix.html


(Marko) #3

I am using logstash-1.4.2


(Marko) #4

Do I have right configuration of zabbix output plugin, as I use logstasth 1.4.2 or i need something to change?


(Aaron Mildenstein) #5

I'm not sure. I rewrote the plugin to use native Ruby for Logstash 1.5.0 because the dependency on zabbix_sender was extremely slow. The old plugin was also not extensible. The pre-1.5 version of the plugin was not terrific. I recommend upgrading Logstash and using the new version of the zabbix plugin.


(Marko) #6

I will upgrade then to logstash 1.5. Could you give me simple conf example of zabbix output plugin for 1.5?
Also do i have to change any other syntax in conf file?

Thanks in advance, I appreciate your help!


(Marko) #7

[quote]I have upgraded to Logstash 1.5
Now my zabbix output looks like:

zabbix {
zabbix_host => "re01os01.net.ot.hr",
zabbix_key => "broj",
zabbix_value => "message"
}
}

but when i start logstash i got error:

Error: Expected one of #, {, } at line 35, column 36 (byte 784) after output {
stdout { codec => rubydebug }

zabbix {
zabbix_host => "re01os01.net.ot.hr"
You may be interested in the '--configtest' flag which you can
use to validate logstash's configuration before you choose
to restart a running system.

I don't get it, help!
[/quote]

Continuing the discussion from Logstash with zabbix output ISSUE:

I found error, it was about ","
Now I start logstash and it says "Logstash startup completed", and when i input example of Log i got message :"Skipping zabbix output; field referenced by re01os01.net.ot.hr is missing {:level=>:warn}"

I have configured re01os01.net.ot.hr as Hostname in Zabbix, and my conf looks like:

output {
zabbix {
zabbix_host => "re01os01.net.ot.hr"
zabbix_key => "broj"
zabbix_value => "message"
}
}


(Aaron Mildenstein) #8

I can't see your entire output block, but it appears that you have an extra closing curly brace there.


(Marko) #9

Here is my complete .conf

input {
tcp {
port => 6000
type => syslog
}
udp {
port => 6000
type => syslog
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:
syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDA
TA:syslog_message}" }
add_tag => [ "zabbix-sender" ]
add_field => ["received_at", "%{@timestamp}"]
add_field => ["received_from", "%{@host}"]
add_field => ["send_field", "%{@message}"]
add_field => [ "zabbix_host", "%{test01}" ]
}
syslog_pri { }

date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}

output {

stdout { codec => rubydebug }

zabbix {
zabbix_host => "test01"
zabbix_key => "Logstash"
zabbix_value => "message"
zabbix_server_host => "10.1.14.205"
}
}

When I telnet to port 6000 and put log message i get:

{
"message" => "Apr 30 01:51:56 test01 3795: Apr 30 01:51:55.941: %BGP-5-ADJCHANGE: neighbor 10.10.1.33 Up \r",
"@version" => "1",
"@timestamp" => "2015-04-29T23:51:56.000Z",
"host" => "0:0:0:0:0:0:0:1",
"type" => "syslog",
"syslog_timestamp" => "Apr 30 01:51:56",
"syslog_hostname" => "test01",
"syslog_program" => "3795",
"syslog_message" => "Apr 30 01:51:55.941: %BGP-5-ADJCHANGE: neighbor 10.10.1.33 Up \r",
"received_at" => "2015-06-03T03:30:02.183Z",
"received_from" => "%{@host}",
"send_field" => "%{@message}",
"zabbix_host" => "%{test01}",
"tags" => [
[0] "zabbix-sender"
],
"syslog_severity_code" => 5,
"syslog_facility_code" => 1,
"syslog_facility" => "user-level",
"syslog_severity" => "notice"
}
Skipping zabbix output; field referenced by test01 is missing {:level=>:warn}


(Aaron Mildenstein) #10

Which version of Logstash are you using? The "send_field" reference suggests 1.4.2 or older.

The version that ships with 1.5 requires you to have certain configuration values in fields, rather than as strings in the config. The config option "zabbix_host" must refer to a field name whose value will be sent as the the zabbix host in the sender packet. The same is true of the zabbix_key and zabbix_value directives.

With the configuration you provided, that implies that field "test01" will contain the value of the zabbix host, the field "Logstash" will contain the zabbix key, and the field "message" will contain the value to send associated with that key and host. Please see the documentation for more details.

The version of the plugin I am working on right now will support multiple fields per event.


(Marko) #11

I am using version 1.5, i deleted send_field as i did not know it's unnecesarry.
You are right, i want to send vaule from field "message" to host "test01", i made zabbix trapper item callet "Logstash" and key is also "Logstash".
So what do i do wrong when i get "Skipping zabbix output; field referenced by test01 is missing {:level=>:warn}"?


(Aaron Mildenstein) #12

The value "test01" must be in a field. In your case, it appears in "syslog_hostname." So your zabbix output block should have:

zabbix {
  zabbix_host => "syslog_hostname"
  ...
}

In this way, the keys are provided programmatically instead of manually.


(Marko) #13

Thanks! Now i modified conf and i think it's OK now, but i still have issues, when i put log i got message:
"Zabbix server at 10.1.14.205 rejected all items sent. "


(Alexander) #14

@Marko,

have you added "allowed host" to your zabbix item?


(Aaron Mildenstein) #15

Indeed, this suggests that item (zabbix_key) does not exist in "host" in zabbix. Logstash will not create the items for you. You need to do this yourself before the item will "arrive" in Zabbix.


(Marko) #16

I did create zabbix trapper item, and that item is applied on my host "test01", and i still get this error.
What should i put into "allowed host" field? My zabbix is on localhost, port is default too...


(Aaron Mildenstein) #17

I don't know what to tell you. That error can only occur if the host & key do not match up. It's the equivalent of using zabbix_sender and seeing:

"response"=>"success", "info"=>"processed 0; Failed 1; Total 1; seconds spent: 0.000018" 

(Okay, that's the ruby version, but it's very close). You sent one item, but it failed to be recognized by the Zabbix server.


(Marko) #18

filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSIN$
add_tag => [ "zabbix-sender" ]
add_field => ["received_at", "%{@timestamp}"]
add_field => ["received_from", "%{host}"]
add_field => ["send_field", "%{message}"]
add_field => [ "zabbix_host", "test01" ]
add_field => [ "zabbix_key", "Logstash" ] }
syslog_pri { }

date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}

And output:

zabbix {
zabbix_host => "zabbix_host"
zabbix_key => "zabbix_key"
zabbix_value => "syslog_message"
zabbix_server_host => "10.1.14.205"
}
}

I really don't understand what's wrong... I even tried to send mannually to trapper:
zabbix_sender -z 10.1.14.205 -p 10051 -s "test01" -k Logstash -o "test value" and i got "test value" message into zabbix...


(Aaron Mildenstein) #19

I don't see a field called syslog_message. Is it there?


(Marko) #20

Yes, i fixed that, it should not be "syslog_message", it should have been "send_field".
Now i corrected that and WHOA! No error when i put log message, BUT i still got nothing in zabbix!
Im gonna kill myself!!! :frowning: