# Logstash instance scaling

**URL:** <https://discuss.elastic.co/t/logstash-instance-scaling/103497>\
**Category:** Logstash\
**Created:** [October 11, 2017, 8:04am UTC](https://discuss.elastic.co/t/logstash-instance-scaling/103497 "2017-10-11T08:04:11Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ondro\_Tadanai](https://avatars.discourse-cdn.com/v4/letter/o/ce7236/32.png) [@Ondro\_Tadanai](https://discuss.elastic.co/u/Ondro_Tadanai)\
**Post date:** [October 11, 2017, 8:04am UTC](https://discuss.elastic.co/t/logstash-instance-scaling/103497/1 "2017-10-11T08:04:11Z")

</div>

Hi,  
I am using logstash with logstash-twitter-input to scrape some tweets in real-time.  
How to configure logstash scaling, if I want to run 2 instances (for example) how to effectively configure logstash instances to cooperate with each other effectively with twitter-input-plugin?  
Thanks at all  
Ondrej

---

<div class="post-metadata">

**Author:** ![Ondro\_Tadanai](https://avatars.discourse-cdn.com/v4/letter/o/ce7236/32.png) [@Ondro\_Tadanai](https://discuss.elastic.co/u/Ondro_Tadanai)\
**Post date:** [October 14, 2017, 5:55pm UTC](https://discuss.elastic.co/t/logstash-instance-scaling/103497/2 "2017-10-14T17:55:21Z")

</div>

Any help please?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 15, 2017, 7:46pm UTC](https://discuss.elastic.co/t/logstash-instance-scaling/103497/3 "2017-10-15T19:46:24Z")

</div>

Logstash has no built-in support for getting two or more instances to talk to each other. If a single Logstash instance won't process the events fast enough I suggest you use one instance to just read the tweets and do nothing else except pass the events to any number of other instances, preferably using an in-between broker with a common queue for all reading instances.

---

<div class="post-metadata">

**Author:** ![Ondro\_Tadanai](https://avatars.discourse-cdn.com/v4/letter/o/ce7236/32.png) [@Ondro\_Tadanai](https://discuss.elastic.co/u/Ondro_Tadanai)\
**Post date:** [October 18, 2017, 1:52pm UTC](https://discuss.elastic.co/t/logstash-instance-scaling/103497/4 "2017-10-18T13:52:04Z")

</div>

Thanks for reply,  
i have additional question,  
if i have "main" logstash instance just to get tweets and send it to another logstash instances, there is single point of failure in my infrastructure with this "main" logstash.  
If i launch 2 instances with same twitter configuration, how to set up these instances, i am afraid that both will write all tweets to broker (or redis for example), so every tweet will be there twice. How to put in the broker only tweets there are already not there?  
Thanks at all  
Ondrej

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 19, 2017, 10:00am UTC](https://discuss.elastic.co/t/logstash-instance-scaling/103497/5 "2017-10-19T10:00:25Z")

</div>

I would probably

- make all Logstash instances pull tweets and push them to a broker,
- write a small service that fetches from the broker and performs deduplication (i.e. drops messages it has seen before) and posts to another queue, and
- configure Logstash to pull from _that_ queue.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 16, 2017, 10:01am UTC](https://discuss.elastic.co/t/logstash-instance-scaling/103497/6 "2017-11-16T10:01:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.