Logstash 2.1.1 : Routing output to multiple clusters


#1

Hi,
I have a need to route different message types to different elasticsearch clusters. I realise I could do this by having different "elasticsearch" output blocks separated by "if" conditionals. However, that is wasteful and long winded...
What I'd like to do is set a variable containing the list of hosts for the cluster corresponding to that message type. Is this possible? If so, how would you recommend I do this?

Regards,
David


(Christian Dahlqvist) #2

The correct way to do this is using conditionals. A single Elasticsearch output can only send data to a single cluster.


(Thomas Decaux) #3

You could try to use:

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-hosts

With a value from your message.

hosts => ["my-cluster-%{cluster}:9200"]

PS: I didn't try it


(Magnus Bäck) #4

In addition to what other folks have suggested, you can use the translate filter to look up a message type in a table and obtain a hostname. That way you don't have to reload the Logstash configuration when you want to update the mapping table.


#5

Is the translate yaml dictionary file read once or each time the filter is applied?


(Magnus Bäck) #6

The YAML file is periodically read. See the documentation.


(system) #7