Beats deployment with logstash relays to a central LS instance

Elastic Stack Beats
1

Is there a documented way to configure this setup?

beat -> regional LS relay -> central LS -> ES

I'd like to minimize and control the number of connections to my central ES cluster via an ACL, and trying to maintain an ACL for every system with a beat deployed is unmanageable (think 5K servers at 20 sites).

I've managed to make it work, but it was REALLY ugly. The problem is that metadata isn't passed on to the central LS instance, so the ES output plugin documented in the getting started guide doesn't work. My kludge is a combination of tagging on the regional LS relay where the metadata is available and then using an 'if "topbeat" in [tag]" to hardcode the missing metadata used for the index name (e.g. topbeat-%{+YYYY.MM.dd})

If this isn't something that is available, or planned or there isn't a better way to proxy beats traffic, I'd be happy to submit an enhancement request.

2

Seems like you have an interesting use case here. Some questions here:

  • What is the meta that that is not passed on? Hostname or fields?
  • What is your ACL looking like? Is it just about being able to write in general or also index / type / field specific ACL's?

One issue that is indirectly related to this is about central config management: https://github.com/elastic/libbeat/issues/200
An other topic which we started to discuss internally but didn't get to a conclusion yet, if a "forwarder" beat could be useful. In a way it would exactly do what your regional LS relay seems to be doing at the moment, except that it would not be able to use grok. But we should be able to set the meta data correct at the senders level.

3
  • It appears that all metadata from the remote forwarder is missing at the central LS instance. I tested this by creating the following output module:

file { path => "/tmp/quux" message_format => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" }

Which should log something like "topbeat-2015.12.01", which is does on the remote forwarder LS instance. On the central LS instance, it logs '%{[@metadata][beat]}-2015.12.01'. Running the central LS in debug mode shows LS trying to use the ES index '-2015.12.01'

  • I should have been more clear when I spoke about ACLs. I was speaking about network and host based firewalls, not ES ACLs. Either way, far too many to manage.

In all honesty, a forwarder beat would solve the problem.

4

So the main problem is the '@metadata' field being discarded on send? Which is the default behavior in logstash (@metadata is discarded on purpose). Which plugin are you using for connecting the logstash instances. Maybe it's worth a config in Logstash to also forward @metadata for cases of Logstash 'chains' being connected either directly or via some intermediate queuing instance (like redis, kafka).

Instead of working with tags you can use the mutate filter to copy required @metadata into the event (in input logstash instance):

filter {
  mutate {
   add_field {
     "beat" => "%{[@metadata][beat]}"
   }
  }
}

I think the type is already part of the event, otherwise use add_field to also copy @metadata.type.

It's up to you if you want to remove the "beat" field in central LS again (also doable using mutate filter). But assuming you don't want to remove "beat" field from event you can configure the elasticsearch output like this:

output {
  elasticsearch {
    index => "%{beat}-%{+YYYY.MM.dd}"
    document_type => "%{type}"
    ...
  }
}

No need to hardcode "topbeat" and/or working with tags plus multiple conditions. You can also try to reconstruct '@metadata' in central LS, by first copying '@metadata' into 'metadata' in event using the mutate filter.

1 Like
5

That is far more elegant than what I have in place. I guess I beat my head on this so much that I killed too many brain cells.

That should solve my immediate need, but a forwarding beat would be generally useful.

I think I'll spend some time with haproxy and see if it will work.

Thanks for putting me back on the right track.