# Performance split for logstash and s3

**URL:** <https://discuss.elastic.co/t/performance-split-for-logstash-and-s3/230260>\
**Category:** Logstash\
**Created:** [April 28, 2020, 10:13pm UTC](https://discuss.elastic.co/t/performance-split-for-logstash-and-s3/230260 "2020-04-28T22:13:05Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sc5283](https://avatars.discourse-cdn.com/v4/letter/s/97f17d/32.png) [@sc5283](https://discuss.elastic.co/u/sc5283)\
**Post date:** [April 28, 2020, 10:13pm UTC](https://discuss.elastic.co/t/performance-split-for-logstash-and-s3/230260/1 "2020-04-28T22:13:06Z")

</div>

Is there a way to split the load on multiple logstash instances to grab data from S3?  
example:  
I would like for logstash instance 1 to grab only files ending in 1 and 2  
logstash instance 2 to grab files ending in 3 and 4  
and so on.  
S3 looks like:  
bucket/2020/04/26/20/file001.gz  
bucket/2020/04/26/20/file002.gz  
bucket/2020/04/26/21/file003.gz  
bucket/2020/04/27/01/file004.gz  
bucket/2020/04/27/03/file005.gz  
bucket/2020/04/28/18/file006.gz  
bucket/2020/04/28/19/file007.gz  
bucket/2020/04/28/20/file008.gz  
....

is there a specific configuration option in logstash that allows for this?  
or maybe a different plug-in? or solution?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 28, 2020, 10:22pm UTC](https://discuss.elastic.co/t/performance-split-for-logstash-and-s3/230260/2 "2020-04-28T22:22:17Z")

</div>

I do not think that is possible. You could use different prefixes (bucket/2020/04/2, bucket/2020/04/1, etc.) but you cannot use a regexp.

---

<div class="post-metadata">

**Author:** ![Luca\_Belluccini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luca_belluccini/32/33239_2.png) [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)\
**Post date:** [April 28, 2020, 10:25pm UTC](https://discuss.elastic.co/t/performance-split-for-logstash-and-s3/230260/3 "2020-04-28T22:25:14Z")

</div>

Hello @sc5283

It is not possible to filter by the suffix of the files, but as @Badger says, it is possible to partition the inputs using the `bucket` name and a `prefix` within the bucket.

All the options are shown in [our doc](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-s3.html#plugins-inputs-s3-prefix).

* * *

Alternatives might be:

- Using different buckets, one input per bucket with Logstash S3 Input
- Using one bucket and different prefixes, one input per bucket & prefix with Logstash S3 Input
- Enable SQS and use Logstash S3 SNS SQS Input ([https://github.com/cherweg/logstash-input-s3-sns-sqs](https://github.com/cherweg/logstash-input-s3-sns-sqs))
- Enable SQS and use Filebeat with S3 Input ([https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-s3.html#filebeat-input-s3](https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-s3.html#filebeat-input-s3))
- Enable SQS and use Functionbeat ([https://www.elastic.co/guide/en/beats/functionbeat/master/configuration-functionbeat-options.html#configuration-functionbeat-options](https://www.elastic.co/guide/en/beats/functionbeat/master/configuration-functionbeat-options.html#configuration-functionbeat-options))

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 26, 2020, 10:25pm UTC](https://discuss.elastic.co/t/performance-split-for-logstash-and-s3/230260/4 "2020-05-26T22:25:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.