How to auto-set zabbix_host?

Elastic Stack Logstash
1

I'm trying to set up a logstash-> zabbix gateway, that collects inputs from multiple servers running filebeat.

I gather that zabbix_host is a mandatory field.
It also seems like it would be set to the ACTUAL generating host, not the one running the logstash server. So that zabbix can fire off alerts about the actual relevant host.

How can I do this?

After much searching, I stumbled across a reference to using

zabbix_host => "%{source_host}"

but apparently that isnt valid either.
What can I do here? Kinda in shock that I cant seem to find any working examples of this.

Logstash 7.5

2

Is this for the zabbix output plugin? We don't use that plugin, but we use both ELK and Zabbix. I suspect zabbix_host would be the case sensitive host name as defined to zabbix. The agent.hostname might work, depending if it is FQDN and if you used that in Zabbix.

I think the syntax would be "%{[agent][hostname]}".

3

Hmm... "%{[agent][hostname]}" sounds potentially useful.
I was also thinking maybe [@metadata][host] ?
Is there any listof all the predefined @metadata or [agent] type variables somewhere?
I've had no luck finding those either :frowning:

4

Ha! I looked for a list of metadata items while I was writing the first reply and couldn't find it either.

You could add a stdout output and print the event in json debug format to see all the available fields.

5

actually i'm not sure stdout shows you metadata fields.

6

No, it doesn't. The agent fields are listed in the filebeat modules section under "beats".

7

Pardon?
I looked at
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules.html
but didnt see what you mean

8

Oops, it's the section after modules, this should be it.

9

Aha.

Filebeat Reference [master] » Exported fields » Beat fields

Thanks.

Might you have any insight into the [@metadata] fields as well?

10

Hmm. Also..
the doc you referenced, mentioned
" beat.name
type: alias
alias to: host.name"

where does this "host.name" come from?
(which is presumably %[host][name] ?)

that sounds like the same thing, but even more standardized. Would be good to understand that I think.

rather important, because when I do a file dump,
i see
"beat":{"hostname":

and

"host":{"name":

but I dont see any mention of "agent".

11 
output { stdout { codec => rubydebug { metadata => true } } }

will print the [@metadata] along with the event.

12

These fields are being renamed for ECS "common schema".... A noble goal but a clear violation of "if it's not broke don't fix it". :slight_smile:

I just thought there were noting that these fields are aliased in the template for used in Elasticsearch, I think you only see one in the document when it hits logstash.

13

Hit some major weirdness.
I used that very useful ruby debug output thing.
determine that the metadata is very small. ONLY actually has:
type
beat
ip_address

So I tried using

zabbix {
zabbix_server_host => "x.x.x.x"
zabbix_host => "%{[@metadata][ip_address]}"
zabbix_key => "filebeat.XXXX"
}

But I get handed
org.jruby.exceptions.RuntimeError: (RuntimeError) Invalid FieldReference: %{[@metadata][ip_address]}

SO I tried using instead,
zabbix_host => "%{[beat][hostname]}"

but get more or less the same error.

Which is really wierd, because I CAN use

            file {
                    path => "/var/log/logstash/beattest-%{[beat][hostname]}.debug"
                    flush_interval => 1
            }

whats the difference??

Dont think it will help any, but here is the FULL trace from the logfile

[2020-01-08T10:12:50,489][FATAL][logstash.runner ] An unexpected error occurred! {:error=>java.lang.IllegalStateException: org.jruby.exceptions.RuntimeError: (RuntimeError) Invalid FieldReference: %{[beat][hostname]}, :backtrace=>["org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:85)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:440)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:304)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:251)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:745)"]}

14

Loooking at the code, it is not using event.sprintf, as any normal plugin would. It is just doing an event.get on it. So you might find that

zabbix_host => "[@metadata][ip_address]"

works.

15

I think I tried that.
or at least, I tried "[beat][hostname]"

here's the really weird thing:
Neither of

                      zabbix_host => "[beat][hostname]"
  or
                      zabbix_host => "%{[beat][hostname]}"

seemed to work. but

 filter {
    mutate {
            add_field => { "[@metadata][zabbix_host]" => "%{[beat][hostname]}" }
    }
 }

 zabbix {
            zabbix_host => "[@metadata][zabbix_host]"
 }

kinda worked.

the ONE thing that is now stopping me from having a fully working zabbix output, is

[2020-01-08T11:20:17,173][WARN ][logstash.outputs.zabbix ][main] Field referenced by filebeat.XXXX is missing
[2020-01-08T11:20:17,431][WARN ][logstash.outputs.zabbix ][main] Zabbix server at x.x.x.x rejected all items sent. {:zabbix_host=>"xxxxxxx"}

Is this saying that I cant just do
zabbix_key => "filebeat.XXXX"

I have to do another stupid indirect reference for [@metadata][zabbix_key] or something??
What kind of whackjob plugin is this??
:-/

16

That is weird. Is it possible you have removed the [beat] field by the time the event gets to the output?

17

This is how I got it to work all the way to zabbix.
Ugh, stupid buggy zabbix module.

filter {
    mutate {
        add_field => { "[@metadata][zabbix_host]" => "%{[beat][hostname]}" }
    }
    if "magic-string-we-care-about" in [message] {
            mutate {
                    add_field => { "[@metadata][zabbix_key]"   => "filebeat.OUR_UNIQUEKEYHERE" }
                    add_field => { "[@metadata][sendtozabbix]" => "yes" }
            }
    }
}
output {
    if [@metadata][sendtozabbix] == "yes" {
            zabbix {
                    zabbix_server_host => "x.x.x.x"
                    zabbix_host => "[@metadata][zabbix_host]"
                    zabbix_key  => "[@metadata][zabbix_key]"
            }
    }
}
18

Is it possible you have removed the [beat] field by the time the event gets to the output?

PS to badger: no. I checked by doing a dump-to-file of the message in parallel in the output section, after the attempt to send to zabbix.

19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.