# How does logstash route the data to a NEW primary ES node, not the old one? **URL:** \ **Category:** Logstash\ **Created:** [July 22, 2018, 12:18pm UTC](https://discuss.elastic.co/t/how-does-logstash-route-the-data-to-a-new-primary-es-node-not-the-old-one/141030 "2018-07-22T12:18:30Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Jacob\_Smith](https://avatars.discourse-cdn.com/v4/letter/j/71e660/32.png) [@Jacob\_Smith](https://discuss.elastic.co/u/Jacob_Smith)\ **Post date:** [July 22, 2018, 12:18pm UTC](https://discuss.elastic.co/t/how-does-logstash-route-the-data-to-a-new-primary-es-node-not-the-old-one/141030/1 "2018-07-22T12:18:30Z")
I'm trying to wrap my head around something. I have 3 ES nodes, with just 1 shard for simplicity. The first node is just the master node. The second node holds the primary shard and the third one holds the replica shard. Master Node: 10.42.0.100:9200 Data node1 (Primary): 10.42.0.101:9200 Data node2 (Replica): 10.42.0.102:9200 This is my config from logstash, where I write the data: ``` output { elasticsearch { hosts => ["10.42.0.101:9200"] index => "twitter" document_type => "tweet" template => "/etc/logstash/template/twitter_template.json" template_name => "twitter" } } ``` Everything looks good and logstash will write the data to my primary ES node. However - what if that node completely dies? How do I make it failover and write to the replica node? According the [elastic.co](http://elastic.co) documentation, the master node keeps track of all this and will assign a new primary node if something goes wrong. However, my logstash config doesn't know this since it's hardcoded to the first node. How can I notify logstash that the primary is down and a new one has been assigned? First, I was thinking of this kind of configuration. ``` output { elasticsearch { hosts => ["10.42.0.101:9200", "10.42.0.102:9200"] index => "twitter" document_type => "tweet" template => "/etc/logstash/template/twitter_template.json" template_name => "twitter" } } ``` Writing data to both the replica and primary - but this is just wrong right? The primary already replicates data to the second node so it doesn't make any sense to write to them both. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 22, 2018, 1:03pm UTC](https://discuss.elastic.co/t/how-does-logstash-route-the-data-to-a-new-primary-es-node-not-the-old-one/141030/2 "2018-07-22T13:03:16Z")
> [@Jacob\_Smith](#): > > The primary already replicates data to the second node so it doesn't make any sense to write to them both. It does not write to both -- "If given an array it will load balance requests across the hosts specified in the hosts parameter." ---
**Author:** ![Jacob\_Smith](https://avatars.discourse-cdn.com/v4/letter/j/71e660/32.png) [@Jacob\_Smith](https://discuss.elastic.co/u/Jacob_Smith)\ **Post date:** [July 22, 2018, 1:07pm UTC](https://discuss.elastic.co/t/how-does-logstash-route-the-data-to-a-new-primary-es-node-not-the-old-one/141030/3 "2018-07-22T13:07:27Z")
Ok, but either way - is writing data to both the replica and primary the way to go here - in order to be fault tolerant? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 22, 2018, 3:17pm UTC](https://discuss.elastic.co/t/how-does-logstash-route-the-data-to-a-new-primary-es-node-not-the-old-one/141030/4 "2018-07-22T15:17:46Z")
> Ok, but either way - is writing data to both the replica and primary the way to go here - in order to be fault tolerant? Again, saying "writing to both" is a misnomer since that's not what happens. But yes, list all known ES nodes in the elasticsearch output (and consider enabling the `sniffing` option) so that Logstash sends requests to any available node and lets the ES cluster figure out which node has the primary shard for each document that's to be stored. (For clusters sufficiently big to have master-only nodes it's a good idea to avoid those nodes.) ---
**Author:** ![Jacob\_Smith](https://avatars.discourse-cdn.com/v4/letter/j/71e660/32.png) [@Jacob\_Smith](https://discuss.elastic.co/u/Jacob_Smith)\ **Post date:** [July 22, 2018, 4:18pm UTC](https://discuss.elastic.co/t/how-does-logstash-route-the-data-to-a-new-primary-es-node-not-the-old-one/141030/5 "2018-07-22T16:18:34Z")
Ok, thanks. So I should list all data nodes - including the replicas? Is it possible to solely enable the sniffing option and leave the hosts parameter blank, since the sniffing adds them to the hosts list anyway? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 22, 2018, 5:04pm UTC](https://discuss.elastic.co/t/how-does-logstash-route-the-data-to-a-new-primary-es-node-not-the-old-one/141030/6 "2018-07-22T17:04:34Z")
Yes. Think about what happens if 10.42.0.101 crashes. The replica gets promoted to primary and everything should keep on running. If you do not include 10.42.0.102 you will not be able to failover. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 22, 2018, 5:32pm UTC](https://discuss.elastic.co/t/how-does-logstash-route-the-data-to-a-new-primary-es-node-not-the-old-one/141030/7 "2018-07-22T17:32:55Z")
> So I should list all data nodes - including the replicas? Replica _nodes_ do not exist. _Shards_ have primaries and (possibly) replicas. The shard a particular document ends up in is entirely determined by ES and is not observable from Logstash. > Is it possible to solely enable the sniffing option and leave the hosts parameter blank, since the sniffing adds them to the hosts list anyway? How would the sniffing code know which ES host to contact in the first place? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 19, 2018, 5:32pm UTC](https://discuss.elastic.co/t/how-does-logstash-route-the-data-to-a-new-primary-es-node-not-the-old-one/141030/8 "2018-08-19T17:32:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.