Beats deployment with logstash relays to a central LS instance

Elastic Stack Beats
4

So the main problem is the '@metadata' field being discarded on send? Which is the default behavior in logstash (@metadata is discarded on purpose). Which plugin are you using for connecting the logstash instances. Maybe it's worth a config in Logstash to also forward @metadata for cases of Logstash 'chains' being connected either directly or via some intermediate queuing instance (like redis, kafka).

Instead of working with tags you can use the mutate filter to copy required @metadata into the event (in input logstash instance):

filter {
  mutate {
   add_field {
     "beat" => "%{[@metadata][beat]}"
   }
  }
}

I think the type is already part of the event, otherwise use add_field to also copy @metadata.type.

It's up to you if you want to remove the "beat" field in central LS again (also doable using mutate filter). But assuming you don't want to remove "beat" field from event you can configure the elasticsearch output like this:

output {
  elasticsearch {
    index => "%{beat}-%{+YYYY.MM.dd}"
    document_type => "%{type}"
    ...
  }
}

No need to hardcode "topbeat" and/or working with tags plus multiple conditions. You can also try to reconstruct '@metadata' in central LS, by first copying '@metadata' into 'metadata' in event using the mutate filter.

1 Like