Sending logstash data to multiple elastic instances/clusters

Elastic Stack Logstash
1

Is it possible to send same data records/events from Logstash output section to two different elastic targets - one 2-node cluster and another separate elastic instance ? Second target would be for testing - so that same data feed is used for that.

Thanks

2

Yep, just have multiple outputs for the events.

3

Great - thank you

4

.. forgot to ask - if there are two outputs - one for each elastic target - what if one of those is down? Will that stop the data from getting into the other one - is it 'all-or-none' output from Logstash or will events still flow to the available elastic cluster if the other one is down?

5

It's an all or nothing thing.

6

Got it - thank you.

7

I would like to reopen this question - with 5 x ES, did this change? All or none is not practical when feeding same data to prod and non-prod (for testing/QA for example) since the availability is different.

If not, could that be done at filebeat level?

Thanks

8

You can have multiple output configs.

You can then use IF statements to decided what gets to the output target.

Mine looks like this:

output {
  if [pipeline] =~ /(?i)(####|####)/ {
#   elasticsearch {
#     hosts => ["http://x.x.x.x:9200","http://x.x.x.x:9200","http://x.x.x.x:9200","http://x.x.x.x:9200"]
#     index => "logstash-%{+YYYY.MM.dd}"
#   }
  }
}

I have another one, with the same setup, just different values in the IF statement.

9

Thank you, I am assuming that within one 'hosts' line all the listed target hosts are all-or-none? Those 4 you listed in this IF ... Then using multiple IF blocks one could more flexibly control different target groups, that are not necessary equally available ...

Thank you

10

If you define multiple targets within a single output, the data shipped off is round robin load balanced. In my case, that config will send log A to the first IP, log B, to the second IP, log C to the third IP, and log D to the forth IP. Then log E will go back to the first IP.

If you want the same data, in two places, you have to define two separate outputs.

11

Thank you Jason!

12

Not a problem.