# A question around logstash S3 input plugin

**URL:** <https://discuss.elastic.co/t/a-question-around-logstash-s3-input-plugin/344769>\
**Category:** Logstash\
**Created:** [October 10, 2023, 11:37pm UTC](https://discuss.elastic.co/t/a-question-around-logstash-s3-input-plugin/344769 "2023-10-10T23:37:48Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![pk.241011](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pk.241011/32/86285_2.png) [@pk.241011](https://discuss.elastic.co/u/pk.241011)\
**Post date:** [October 10, 2023, 11:37pm UTC](https://discuss.elastic.co/t/a-question-around-logstash-s3-input-plugin/344769/1 "2023-10-10T23:37:48Z")

</div>

Hi All,

We run logstash on multiple EC2 instances behind a loadbalancer for reliability purposes. We are thinking of using the S3 input plugin. Since the servers are created by auto-scaling process of AWS, they are exactly same.

I am trying to get some clarity around the behaviour of the S3 input plugin when multiple instances of Logstash are running polling the same S3 bucket and prefix.

My understanding is that they should be fine since each S3 object is key looks like a filepath but is not actually a filepath.

Sample code for my case will be like this. I delete the S3 object after reading it. So no need to track the last handled file.

```auto
input
{
	s3
	{
		bucket => "testbucket"
		prefix => "get/this/data"
		region => "us-east-2"
		delete => true
		interval => 100
		sincedb_path => "/dev/null"
		additional_settings => {
			"force_path_style" => true
			"follow_redirects" => false
			}
	}
}

```

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [October 11, 2023, 12:31am UTC](https://discuss.elastic.co/t/a-question-around-logstash-s3-input-plugin/344769/2 "2023-10-11T00:31:19Z")

</div>

> [@pk.241011](#):
>
> I am trying to get some clarity around the behaviour of the S3 input plugin when multiple instances of Logstash are running polling the same S3 bucket and prefix.

This input does not support this, it can lead to duplicates as you cannot guarantee that multiple Logstash instances will not try to read the same object in S3 at the same type.

If you need to have multiple instances reading the same bucket you should something that support it, one option is to use Filebeat with the [AWS S3 Input with SQS](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-aws-s3.html#_parallel_processing) configured, this is also the recommend way to consume logs from S3 buckets.

---

<div class="post-metadata">

**Author:** ![pk.241011](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pk.241011/32/86285_2.png) [@pk.241011](https://discuss.elastic.co/u/pk.241011)\
**Post date:** [October 11, 2023, 1:04am UTC](https://discuss.elastic.co/t/a-question-around-logstash-s3-input-plugin/344769/3 "2023-10-11T01:04:04Z")

</div>

Thanks. Makes things a lot clearer. I will go through the link you posted.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 8, 2023, 1:04am UTC](https://discuss.elastic.co/t/a-question-around-logstash-s3-input-plugin/344769/4 "2023-11-08T01:04:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.