# Multiple logstash instances getting data from one redis key?

**URL:** <https://discuss.elastic.co/t/multiple-logstash-instances-getting-data-from-one-redis-key/127970>\
**Category:** Logstash\
**Created:** [April 13, 2018, 12:11pm UTC](https://discuss.elastic.co/t/multiple-logstash-instances-getting-data-from-one-redis-key/127970 "2018-04-13T12:11:14Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\
**Post date:** [April 13, 2018, 12:11pm UTC](https://discuss.elastic.co/t/multiple-logstash-instances-getting-data-from-one-redis-key/127970/1 "2018-04-13T12:11:14Z")

</div>

Hi there,

I changed my setting from filebeat -\> logstash to filebeat -\> redis -\> multiple logstash.

Since we are still in migration from 5.1.2 to 6.2.3 we are running both stacks in parallel.  
No we noticed that we have different document counts on both stacks.

We are not giving an explicit document key, so logstash is generating a uuid for each document. So it would be possible to index the same data multiple times without errors.

Is it save to have a logstash pipline which is running with multiple workers on a redis key?  
And is it save to to run this pipeline in multiple logstash instances?

Or can it happen that different logstash instances are getting the same data from redis and so they produce duplicates against ES?

Currently I am trying to find out, if I have duplicates already in ES or not. But I would like to have your knowledge if I was to naive to do as mentioned above or not.

Thanks, Andreas

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 15, 2018, 5:48pm UTC](https://discuss.elastic.co/t/multiple-logstash-instances-getting-data-from-one-redis-key/127970/2 "2018-04-15T17:48:44Z")

</div>

What kind of Redis data structure are you using (the redis input's `data_type` option)?

---

<div class="post-metadata">

**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\
**Post date:** [April 16, 2018, 7:39am UTC](https://discuss.elastic.co/t/multiple-logstash-instances-getting-data-from-one-redis-key/127970/3 "2018-04-16T07:39:35Z")

</div>

data\_type is **list**

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 16, 2018, 7:51am UTC](https://discuss.elastic.co/t/multiple-logstash-instances-getting-data-from-one-redis-key/127970/4 "2018-04-16T07:51:49Z")

</div>

Redis guarantees that an item in a list can only be popped once so it's safe to have multiple workers reading from the same list.

---

<div class="post-metadata">

**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\
**Post date:** [April 16, 2018, 9:21am UTC](https://discuss.elastic.co/t/multiple-logstash-instances-getting-data-from-one-redis-key/127970/5 "2018-04-16T09:21:11Z")

</div>

ok, great. Good to know. Thanks a lot.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 14, 2018, 9:21am UTC](https://discuss.elastic.co/t/multiple-logstash-instances-getting-data-from-one-redis-key/127970/6 "2018-05-14T09:21:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.