Logstash with zabbix output ISSUE

Cenzoooo · May 28, 2015, 12:21am

Hello!

I have problem, my zabbix output just don't work and i can't find reason why.

For example, i've created simple logstash conf file:

input {
tcp {
port => 6000
type => syslog
}
udp {
port => 6000
type => syslog
}
}
filter {
grep {
type => "linux-syslog"
match => [ "@message", "(error|ERROR|CRITICAL)" ]
add_tag => [ "zabbix-sender" ]
add_field => [
"zabbix_host", "%{@test01}",
"zabbix_item", "Test.broj"
]
}
}

output {
stdout { codec => rubydebug }
zabbix {
tags => "zabbix-sender"
zabbix_sender => "/usr/local/bin/zabbix_sender"
}
}

Then i created Host in zabbix named re01os01.net.ot.hr, zabbix trapper item named "Test" with key "broj" and type of information as "TEXT".

I start logstash (it started well with no errors)
I telnet to localhost port 6000 and type "CRITICAL"
Logstash return me parsed log:
{
"message" => "CRITICAL\r",
"@version" => "1",
"@timestamp" => "2015-05-28T00:08:34.282Z",
"host" => "0:0:0:0:0:0:0:1:50210",
"type" => "syslog"
}

So it works great!

After that, i go to zabbix and check trapper item and it just don't get any value.

Please help!

Best Regards,

Marko

theuntergeek · May 28, 2015, 12:34am

What version of Logstash are you using? The configuration you're using here seems to indicate you're using Logstash 1.4.x or older. If you're using Logstash 1.5, the Zabbix output plugin is totally changed.

See https://www.elastic.co/guide/en/logstash/current/plugins-outputs-zabbix.html

Cenzoooo · May 28, 2015, 5:38am

I am using logstash-1.4.2

Cenzoooo · May 29, 2015, 7:12pm

Do I have right configuration of zabbix output plugin, as I use logstasth 1.4.2 or i need something to change?

theuntergeek · May 29, 2015, 7:28pm

I'm not sure. I rewrote the plugin to use native Ruby for Logstash 1.5.0 because the dependency on zabbix_sender was extremely slow. The old plugin was also not extensible. The pre-1.5 version of the plugin was not terrific. I recommend upgrading Logstash and using the new version of the zabbix plugin.

Cenzoooo · May 29, 2015, 8:24pm

I will upgrade then to logstash 1.5. Could you give me simple conf example of zabbix output plugin for 1.5?
Also do i have to change any other syntax in conf file?

Thanks in advance, I appreciate your help!

Cenzoooo · May 29, 2015, 9:54pm

[quote]I have upgraded to Logstash 1.5
Now my zabbix output looks like:

zabbix {
zabbix_host => "re01os01.net.ot.hr",
zabbix_key => "broj",
zabbix_value => "message"
}
}

but when i start logstash i got error:

Error: Expected one of #, {, } at line 35, column 36 (byte 784) after output {
stdout { codec => rubydebug }

zabbix {
zabbix_host => "re01os01.net.ot.hr"
You may be interested in the '--configtest' flag which you can
use to validate logstash's configuration before you choose
to restart a running system.

I don't get it, help!
[/quote]

Continuing the discussion from Logstash with zabbix output ISSUE:

I found error, it was about ","
Now I start logstash and it says "Logstash startup completed", and when i input example of Log i got message :"Skipping zabbix output; field referenced by re01os01.net.ot.hr is missing {:level=>:warn}"

I have configured re01os01.net.ot.hr as Hostname in Zabbix, and my conf looks like:

output {
zabbix {
zabbix_host => "re01os01.net.ot.hr"
zabbix_key => "broj"
zabbix_value => "message"
}
}

theuntergeek · June 1, 2015, 5:04pm

I can't see your entire output block, but it appears that you have an extra closing curly brace there.

Cenzoooo · June 3, 2015, 5:52am

Here is my complete .conf

input {
tcp {
port => 6000
type => syslog
}
udp {
port => 6000
type => syslog
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:
syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDA
TA:syslog_message}" }
add_tag => [ "zabbix-sender" ]
add_field => ["received_at", "%{@timestamp}"]
add_field => ["received_from", "%{@host}"]
add_field => ["send_field", "%{@message}"]
add_field => [ "zabbix_host", "%{test01}" ]
}
syslog_pri { }

date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}

output {

stdout { codec => rubydebug }

zabbix {
zabbix_host => "test01"
zabbix_key => "Logstash"
zabbix_value => "message"
zabbix_server_host => "10.1.14.205"
}
}

When I telnet to port 6000 and put log message i get:

{
"message" => "Apr 30 01:51:56 test01 3795: Apr 30 01:51:55.941: %BGP-5-ADJCHANGE: neighbor 10.10.1.33 Up \r",
"@version" => "1",
"@timestamp" => "2015-04-29T23:51:56.000Z",
"host" => "0:0:0:0:0:0:0:1",
"type" => "syslog",
"syslog_timestamp" => "Apr 30 01:51:56",
"syslog_hostname" => "test01",
"syslog_program" => "3795",
"syslog_message" => "Apr 30 01:51:55.941: %BGP-5-ADJCHANGE: neighbor 10.10.1.33 Up \r",
"received_at" => "2015-06-03T03:30:02.183Z",
"received_from" => "%{@host}",
"send_field" => "%{@message}",
"zabbix_host" => "%{test01}",
"tags" => [
[0] "zabbix-sender"
],
"syslog_severity_code" => 5,
"syslog_facility_code" => 1,
"syslog_facility" => "user-level",
"syslog_severity" => "notice"
}
Skipping zabbix output; field referenced by test01 is missing {:level=>:warn}

theuntergeek · June 3, 2015, 5:49pm

Which version of Logstash are you using? The "send_field" reference suggests 1.4.2 or older.

The version that ships with 1.5 requires you to have certain configuration values in fields, rather than as strings in the config. The config option "zabbix_host" must refer to a field name whose value will be sent as the the zabbix host in the sender packet. The same is true of the zabbix_key and zabbix_value directives.

With the configuration you provided, that implies that field "test01" will contain the value of the zabbix host, the field "Logstash" will contain the zabbix key, and the field "message" will contain the value to send associated with that key and host. Please see the documentation for more details.

The version of the plugin I am working on right now will support multiple fields per event.

Cenzoooo · June 5, 2015, 5:49am

I am using version 1.5, i deleted send_field as i did not know it's unnecesarry.
You are right, i want to send vaule from field "message" to host "test01", i made zabbix trapper item callet "Logstash" and key is also "Logstash".
So what do i do wrong when i get "Skipping zabbix output; field referenced by test01 is missing {:level=>:warn}"?

theuntergeek · June 5, 2015, 6:52pm

The value "test01" must be in a field. In your case, it appears in "syslog_hostname." So your zabbix output block should have:

zabbix {
  zabbix_host => "syslog_hostname"
  ...
}

In this way, the keys are provided programmatically instead of manually.

Cenzoooo · June 7, 2015, 5:43am

Thanks! Now i modified conf and i think it's OK now, but i still have issues, when i put log i got message:
"Zabbix server at 10.1.14.205 rejected all items sent. "

MrAV · June 8, 2015, 7:33am

@Marko,

have you added "allowed host" to your zabbix item?

theuntergeek · June 8, 2015, 5:31pm

Indeed, this suggests that item (zabbix_key) does not exist in "host" in zabbix. Logstash will not create the items for you. You need to do this yourself before the item will "arrive" in Zabbix.

Cenzoooo · June 9, 2015, 11:38am

I did create zabbix trapper item, and that item is applied on my host "test01", and i still get this error.
What should i put into "allowed host" field? My zabbix is on localhost, port is default too...

theuntergeek · June 9, 2015, 3:42pm

I don't know what to tell you. That error can only occur if the host & key do not match up. It's the equivalent of using zabbix_sender and seeing:

"response"=>"success", "info"=>"processed 0; Failed 1; Total 1; seconds spent: 0.000018"

(Okay, that's the ruby version, but it's very close). You sent one item, but it failed to be recognized by the Zabbix server.

Cenzoooo · June 9, 2015, 4:53pm

filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSIN$
add_tag => [ "zabbix-sender" ]
add_field => ["received_at", "%{@timestamp}"]
add_field => ["received_from", "%{host}"]
add_field => ["send_field", "%{message}"]
add_field => [ "zabbix_host", "test01" ]
add_field => [ "zabbix_key", "Logstash" ] }
syslog_pri { }

date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}

And output:

zabbix {
zabbix_host => "zabbix_host"
zabbix_key => "zabbix_key"
zabbix_value => "syslog_message"
zabbix_server_host => "10.1.14.205"
}
}

I really don't understand what's wrong... I even tried to send mannually to trapper:
zabbix_sender -z 10.1.14.205 -p 10051 -s "test01" -k Logstash -o "test value" and i got "test value" message into zabbix...

theuntergeek · June 9, 2015, 5:08pm

I don't see a field called syslog_message. Is it there?

Cenzoooo · June 9, 2015, 5:39pm

Yes, i fixed that, it should not be "syslog_message", it should have been "send_field".
Now i corrected that and WHOA! No error when i put log message, BUT i still got nothing in zabbix!
Im gonna kill myself!!!

Topic		Replies	Views
Logstash-output-zabbix (3.0.5) Zabbix Version 4 ready? Logstash	2	408	January 10, 2019
Problem pushing log from logstash to zabbix Logstash	2	936	July 6, 2017
Logstash output zabbix Logstash	4	1040	August 7, 2017
Logstash-output-zabbix does not send to zabbix Logstash	4	637	November 3, 2021
Logstash-output-zabbix (question about multi_value) Logstash	4	1077	October 20, 2017

Logstash with zabbix output ISSUE

Related topics