# Logstash concurrency - managing multiple LS nodes that share the same pipeline configuration **URL:** \ **Category:** Logstash\ **Created:** [October 21, 2019, 11:08am UTC](https://discuss.elastic.co/t/logstash-concurrency-managing-multiple-ls-nodes-that-share-the-same-pipeline-configuration/204446 "2019-10-21T11:08:19Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![AlessandroKP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alessandrokp/32/45312_2.png) [@AlessandroKP](https://discuss.elastic.co/u/AlessandroKP)\ **Post date:** [October 21, 2019, 11:08am UTC](https://discuss.elastic.co/t/logstash-concurrency-managing-multiple-ls-nodes-that-share-the-same-pipeline-configuration/204446/1 "2019-10-21T11:08:19Z")
Hi all, I have a question regarding concurrency management in Logstash. My setup is the following. I have two Logstash nodes that run the same pipeline, i.e. they use the same pipeline configuration. They perform exactly the same scheduled operations: they get their input from a common source, make the same processing and then output the data to the same Elasticsearch cluster. Since a Logstash instance always works in a standalone mode, I would like to avoid that the pipeline is executing the same activity at the same time by both instances. I was thinking of solving this by implementing a "semaphore" index on Elasticsearch. The semaphore implemention would work as follows. In the input section of the Logstash pipeline, I query the semaphore index in ES for any document, with a certain periodicity (by using the elasticsearch input plugin). If the query returns any documents, then the pipeline drops the current event. If the query returns nothing, then the pipeline does the following in the filter section: - creates a semaphore document in the semaphore index - does the usual elaboration by enriching the input event - after all activities have been done, remove the semaphore document Then, it outputs the event to Elasticsearch. This solution seems a bit cumbersome. Do you know if there is a simpler way to do it? Alessandro. ---
**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\ **Post date:** [October 23, 2019, 8:51pm UTC](https://discuss.elastic.co/t/logstash-concurrency-managing-multiple-ls-nodes-that-share-the-same-pipeline-configuration/204446/2 "2019-10-23T20:51:14Z")
You could also define a common "document\_id" and just have each logstash overwrite the document. Then it won't matter which one fires first or second because if Logstash A went first and it writes a document with ID ABC123 then when Logstash B runs and it uses the same document ID of ABC123 it will just overwrite the changes. ---
**Author:** ![ea1987](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ea1987/32/51356_2.png) [@ea1987](https://discuss.elastic.co/u/ea1987)\ **Post date:** [October 25, 2019, 9:47am UTC](https://discuss.elastic.co/t/logstash-concurrency-managing-multiple-ls-nodes-that-share-the-same-pipeline-configuration/204446/3 "2019-10-25T09:47:56Z")
Hi Aquax, that's not what we would like to do. I will try to explain this use case in a different way. Pipeline **check\_agent** : this pipeline has a schedule HTTP input which is used to perform request to a specific external agent. This pipeline is stored on a shared NAS. **Logstash\_instance\_1** (running on host **A** ): running pipeline check\_agent **Logstash\_instance\_2** (running on host **B** ): running pipeline check\_agent Since we would like to avoid external agent overhead due to HTTP requests received from both logstash instances, we implemented a kind of concurrency control of the schedule execution using a custom index with a semaphore document (kept update using a specific \_id) used to sync both pipeline instances execution, like Alessandro explained in the previous post. Is this the best approach or are there other better ones? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 22, 2019, 9:47am UTC](https://discuss.elastic.co/t/logstash-concurrency-managing-multiple-ls-nodes-that-share-the-same-pipeline-configuration/204446/4 "2019-11-22T09:47:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.