# How to run multiple logstash instances for s3 input

**URL:** <https://discuss.elastic.co/t/how-to-run-multiple-logstash-instances-for-s3-input/47624>\
**Category:** Logstash\
**Created:** [April 18, 2016, 8:47am UTC](https://discuss.elastic.co/t/how-to-run-multiple-logstash-instances-for-s3-input/47624 "2016-04-18T08:47:11Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![HugoBang](https://avatars.discourse-cdn.com/v4/letter/h/c89c15/32.png) [@HugoBang](https://discuss.elastic.co/u/HugoBang)\
**Post date:** [April 18, 2016, 8:47am UTC](https://discuss.elastic.co/t/how-to-run-multiple-logstash-instances-for-s3-input/47624/1 "2016-04-18T08:47:11Z")

</div>

Hello.

I am trying to install multiple logstash instances for s3 input but it seems to be impossible because each logstash saves a sincedb file locally and even if the sincedb file is shared between the logstash instances, the same object of s3 may be processed simultaneously by multiple logstash instances.

Do you have any idea?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [April 18, 2016, 9:19am UTC](https://discuss.elastic.co/t/how-to-run-multiple-logstash-instances-for-s3-input/47624/2 "2016-04-18T09:19:37Z")

</div>

What does your input look like?

---

<div class="post-metadata">

**Author:** ![HugoBang](https://avatars.discourse-cdn.com/v4/letter/h/c89c15/32.png) [@HugoBang](https://discuss.elastic.co/u/HugoBang)\
**Post date:** [April 18, 2016, 9:26am UTC](https://discuss.elastic.co/t/how-to-run-multiple-logstash-instances-for-s3-input/47624/3 "2016-04-18T09:26:31Z")

</div>

Here is my input sample.

input {  
s3 {  
access\_key\_id =\> "XXXXXXXXXXXXXXXXXXXXX"  
secret\_access\_key =\> "XXXXXXXXXXXXXXXXXXXXXXX"  
region =\> "XXXXXXXXXX"  
bucket =\> "XXXXXXXX"  
prefix =\> "XXXXX/XXXXX/"  
tags =\> ["XXXXXXXX","XXXXXXX"]  
type =\> "elb"  
sincedb\_path =\> "/var/lib/logstash/.sincedb\_XXXXXXXXXX\_elb"  
temporary\_directory =\> "/tmp/logstash/input\_XXXXXXXXXX\_elb"  
}  
}

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [April 18, 2016, 9:29am UTC](https://discuss.elastic.co/t/how-to-run-multiple-logstash-instances-for-s3-input/47624/4 "2016-04-18T09:29:50Z")

</div>

So just do;

```auto
input {
s3 {
access_key_id => "XXXXXXXXXXXXXXXXXXXXX"
secret_access_key => "XXXXXXXXXXXXXXXXXXXXXXX"
region => "XXXXXXXXXX"
bucket => "XXXXXXXX"
prefix => "XXXXX/XXXXX/"
tags => ["XXXXXXXX","XXXXXXX"]
type => "elb"
sincedb_path => "/var/lib/logstash/.sincedb_XXXXXXXXXX_elb"
temporary_directory => "/tmp/logstash/input_XXXXXXXXXX_elb"
}
s3 {
access_key_id => "YYYY"
secret_access_key => "YYYY"
region => "YYYY"
bucket => "YYYY"
prefix => "YYYY/YYYY/"
tags => ["YYYY","YYYY"]
type => "elb"
sincedb_path => "/var/lib/logstash/.sincedb_YYYY_elb"
temporary_directory => "/tmp/logstash/input_YYYY_elb"
}
}

```

Where YYYY and the XXXXXXXXXXXXXXXXXXXXX values are different.

---

<div class="post-metadata">

**Author:** ![HugoBang](https://avatars.discourse-cdn.com/v4/letter/h/c89c15/32.png) [@HugoBang](https://discuss.elastic.co/u/HugoBang)\
**Post date:** [April 19, 2016, 12:50am UTC](https://discuss.elastic.co/t/how-to-run-multiple-logstash-instances-for-s3-input/47624/5 "2016-04-19T00:50:26Z")

</div>

The multiple instances are autoscaled backend instances of ELB and the instances have a same configuration as below.

> input {  
> beats {  
> ...  
> }

> s3 {  
> ...  
> }

I want them to process same bucket and prefix of s3.

---

<div class="post-metadata">

**Author:** ![astickler](https://avatars.discourse-cdn.com/v4/letter/a/53a042/32.png) [@astickler](https://discuss.elastic.co/u/astickler)\
**Post date:** [January 18, 2017, 3:18pm UTC](https://discuss.elastic.co/t/how-to-run-multiple-logstash-instances-for-s3-input/47624/6 "2017-01-18T15:18:08Z")

</div>

Hi Mark - I think Hugo's problem is that he has a single s3 bucket that he wants to process with multiple logstash instances (i.e. for scalability, as there is a lot of data), and he is finding that each instance will process all s3 bucket files, rather than half each, as they do not share a sincedb.

I am also interested in solving this problem.

Thanks,  
Andrew

---

<div class="post-metadata">

**Author:** ![HugoBang](https://avatars.discourse-cdn.com/v4/letter/h/c89c15/32.png) [@HugoBang](https://discuss.elastic.co/u/HugoBang)\
**Post date:** [March 15, 2017, 5:59am UTC](https://discuss.elastic.co/t/how-to-run-multiple-logstash-instances-for-s3-input/47624/7 "2017-03-15T05:59:57Z")

</div>

Yes, astickler.

This problem is still not solved. ☹

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:27am UTC](https://discuss.elastic.co/t/how-to-run-multiple-logstash-instances-for-s3-input/47624/8 "2017-07-06T04:27:52Z")

</div>