# How the logstash s3 input plugin handle load balance?

**URL:** <https://discuss.elastic.co/t/how-the-logstash-s3-input-plugin-handle-load-balance/87064>\
**Category:** Logstash\
**Created:** [May 25, 2017, 2:48am UTC](https://discuss.elastic.co/t/how-the-logstash-s3-input-plugin-handle-load-balance/87064 "2017-05-25T02:48:41Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![hanks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hanks/32/18204_2.png) [@hanks](https://discuss.elastic.co/u/hanks)\
**Post date:** [May 25, 2017, 2:48am UTC](https://discuss.elastic.co/t/how-the-logstash-s3-input-plugin-handle-load-balance/87064/1 "2017-05-25T02:48:41Z")

</div>

Hi, the question is if I have multi logstash servers and set the same conf to collect the s3 bucket, is the result become duplicate? And how to setup load balance for s3 input plugin? Thank you for your time.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 28, 2017, 6:43pm UTC](https://discuss.elastic.co/t/how-the-logstash-s3-input-plugin-handle-load-balance/87064/2 "2017-05-28T18:43:00Z")

</div>

> Hi, the question is if I have multi logstash servers and set the same conf to collect the s3 bucket, is the result become duplicate?

Yes.

> And how to setup load balance for s3 input plugin?

If you ship everything to Elasticsearch (and not anywhere else) you can set the document id of the events that's stored in ES to a checksum of the event contents so that any duplicate(s) will overwrite the any existing document. However, this is obviously very inefficient. The s3 input plugin or Logstash itself has no clustering support to deal with this situation.

---

<div class="post-metadata">

**Author:** ![hanks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hanks/32/18204_2.png) [@hanks](https://discuss.elastic.co/u/hanks)\
**Post date:** [May 30, 2017, 12:29am UTC](https://discuss.elastic.co/t/how-the-logstash-s3-input-plugin-handle-load-balance/87064/3 "2017-05-30T00:29:32Z")

</div>

Thank you for your reply, I decided to download logs in s3 bucket to local file and then use filebeat to collect it

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 27, 2017, 12:29am UTC](https://discuss.elastic.co/t/how-the-logstash-s3-input-plugin-handle-load-balance/87064/4 "2017-06-27T00:29:44Z")

</div>

Closed on June 27, 2017, 12:29am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.