# If/else within Logstash output plugin **URL:** \ **Category:** Logstash\ **Created:** [June 16, 2019, 11:24am UTC](https://discuss.elastic.co/t/if-else-within-logstash-output-plugin/185965 "2019-06-16T11:24:59Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![g.le](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/g.le/32/22526_2.png) [@g.le](https://discuss.elastic.co/u/g.le)\ **Post date:** [June 16, 2019, 11:24am UTC](https://discuss.elastic.co/t/if-else-within-logstash-output-plugin/185965/1 "2019-06-16T11:24:59Z")
Hello, I'm having a Logstash configuration similar to the below: ``` input { beats { port => 5044 } } filter { clone { clones => ["local-dc"] add_tag => ["cloned"] } } output { if "cloned" in [tags] { elasticsearch { hosts => ["elastic-01:9200"] } } else { elasticsearch { hosts => ["elastic-02:9200"] } } } ``` Assuming elastic-01 is temporarily unreachable, I would expect only the messages tagged with "cloned" to fail. Nevertheless, all messages fail. Is that normal? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 16, 2019, 1:09pm UTC](https://discuss.elastic.co/t/if-else-within-logstash-output-plugin/185965/2 "2019-06-16T13:09:57Z")
Yes. Logstash require all outputs to succeed before the batch is considered complete do that is expected behaviour. ---
**Author:** ![g.le](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/g.le/32/22526_2.png) [@g.le](https://discuss.elastic.co/u/g.le)\ **Post date:** [June 16, 2019, 1:22pm UTC](https://discuss.elastic.co/t/if-else-within-logstash-output-plugin/185965/3 "2019-06-16T13:22:58Z")
@Christian_Dahlqvist: Thank you for your feedback on this! Any workarounds (besides using distinct Logstash instances per Elasticsearch destination) ? ---
**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Post date:** [June 16, 2019, 3:02pm UTC](https://discuss.elastic.co/t/if-else-within-logstash-output-plugin/185965/4 "2019-06-16T15:02:27Z")
What are you trying to accomplish? If you want to clone events and send them to the same server, just send them to a different index in the elasticsearch output. ---
**Author:** ![g.le](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/g.le/32/22526_2.png) [@g.le](https://discuss.elastic.co/u/g.le)\ **Post date:** [June 16, 2019, 3:16pm UTC](https://discuss.elastic.co/t/if-else-within-logstash-output-plugin/185965/5 "2019-06-16T15:16:06Z")
What I'm trying to accomplish is to send logs to two distinct Elasticsearch clusters. If that requires two distinct Logstash instances, then so be it. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 16, 2019, 3:41pm UTC](https://discuss.elastic.co/t/if-else-within-logstash-output-plugin/185965/6 "2019-06-16T15:41:30Z")
You can have [two different output pipelines](https://www.elastic.co/guide/en/logstash/7.1/pipeline-to-pipeline.html) (distributor pattern) within a single Logstash instance, both backed by [separate persistent queues](https://www.elastic.co/guide/en/logstash/7.1/persistent-queues.html). ---
**Author:** ![g.le](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/g.le/32/22526_2.png) [@g.le](https://discuss.elastic.co/u/g.le)\ **Post date:** [June 16, 2019, 4:12pm UTC](https://discuss.elastic.co/t/if-else-within-logstash-output-plugin/185965/7 "2019-06-16T16:12:05Z")
@Christian_Dahlqvist: That is an interesting approach (albeit a beta feature). What is not clear from the documents though is whether all filtering now needs to take place within the contents of config/pipelines.yml. Do we need to move filtering logic away from the .conf file? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 16, 2019, 4:14pm UTC](https://discuss.elastic.co/t/if-else-within-logstash-output-plugin/185965/8 "2019-06-16T16:14:09Z")
The pipelines feature is just a different way of organizing your config. ---
**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Post date:** [June 16, 2019, 5:09pm UTC](https://discuss.elastic.co/t/if-else-within-logstash-output-plugin/185965/9 "2019-06-16T17:09:03Z")
> [@Christian\_Dahlqvist](#): > > Yes. Logstash require all outputs to succeed before the batch is considered complete do that is expected behaviour. That seems like an odd design decision, what's the rationale behind it? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 16, 2019, 6:04pm UTC](https://discuss.elastic.co/t/if-else-within-logstash-output-plugin/185965/10 "2019-06-16T18:04:29Z")
It is designed to prevent data loss. If not all outputs were required to succeed any of them could fail and drop data at any time. ---
**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Post date:** [June 18, 2019, 12:00am UTC](https://discuss.elastic.co/t/if-else-within-logstash-output-plugin/185965/11 "2019-06-18T00:00:16Z")
Wouldn't that be where DLQ comes in to pick up failed entries? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 18, 2019, 12:19am UTC](https://discuss.elastic.co/t/if-else-within-logstash-output-plugin/185965/12 "2019-06-18T00:19:52Z")
That would depend on why it failed, I believe. DLQ does not queue everything that fails. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 18, 2019, 4:58am UTC](https://discuss.elastic.co/t/if-else-within-logstash-output-plugin/185965/13 "2019-06-18T04:58:28Z")
DLQ is only supported by the Elasticsearch output plugin as far as I know and only queues documents where Elasticsearch reported an error, not when Elasticsearch was not available. ---
**Author:** ![g.le](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/g.le/32/22526_2.png) [@g.le](https://discuss.elastic.co/u/g.le)\ **Post date:** [June 19, 2019, 4:33pm UTC](https://discuss.elastic.co/t/if-else-within-logstash-output-plugin/185965/14 "2019-06-19T16:33:46Z")
Thanks everyone for the responses. Given that I had a spare Logstash server doing nothing, I ended up duplicating everything: - Two Filebeat instances residing on the host that produces the logs. - Each Filebeat instance ships logs to a dedicated Logstash node. - Each Logstash node pushes documents to a dedicated Elasticsearch cluster. Not the most elegant approach, but it was easy to configure without spending much time converting an existing (and huge) Logstash configuration into pipelines. The regression test would have lasted weeks for no reason. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 17, 2019, 4:33pm UTC](https://discuss.elastic.co/t/if-else-within-logstash-output-plugin/185965/15 "2019-07-17T16:33:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.