# \[s3 Logstash\] Setting bucket prefix to be specific range of days

**URL:** <https://discuss.elastic.co/t/s3-logstash-setting-bucket-prefix-to-be-specific-range-of-days/243787>\
**Category:** Logstash\
**Created:** [August 4, 2020, 9:29pm UTC](https://discuss.elastic.co/t/s3-logstash-setting-bucket-prefix-to-be-specific-range-of-days/243787 "2020-08-04T21:29:49Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ian2](https://avatars.discourse-cdn.com/v4/letter/i/dec6dc/32.png) [@ian2](https://discuss.elastic.co/u/ian2)\
**Post date:** [August 4, 2020, 9:29pm UTC](https://discuss.elastic.co/t/s3-logstash-setting-bucket-prefix-to-be-specific-range-of-days/243787/1 "2020-08-04T21:29:49Z")

</div>

We're pushing logs into an s3 bucket. We've setup the our indexer configuration and tested with a single folder within our bucket - which naming convention is the date (e.g. my-bucket-of-logs/20200804) however we want to be able to pull in logs from the last 14 days (14 folders)

- /20200804, /20200803, /20200802, .......

```auto
input {
	s3 {
		bucket => "my-bucket-of-logs"
		access_key_id => "{{ access_key_id }}"
		secret_access_key => "{{ secret_key_id }}"
		prefix => "/20200804/"
	}
}

```

From the docs =\> [https://www.elastic.co/guide/en/logstash/current/plugins-inputs-s3.html#plugins-inputs-s3-prefix](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-s3.html#plugins-inputs-s3-prefix) prefix is only allowed to be a string so I'm not sure if theres another way around this issue?

Any help or insight would be appreciated.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 4, 2020, 10:17pm UTC](https://discuss.elastic.co/t/s3-logstash-setting-bucket-prefix-to-be-specific-range-of-days/243787/2 "2020-08-04T22:17:42Z")

</div>

You could use 14 inputs in 14 configuration files. Once a day replace the oldest configuration file with one for today. If logstash is running with -r it will reload the pipeline when it sees one of the configuration files change. It's a hack, but it might do what you want to get done.

---

<div class="post-metadata">

**Author:** ![ian2](https://avatars.discourse-cdn.com/v4/letter/i/dec6dc/32.png) [@ian2](https://discuss.elastic.co/u/ian2)\
**Post date:** [August 5, 2020, 7:37pm UTC](https://discuss.elastic.co/t/s3-logstash-setting-bucket-prefix-to-be-specific-range-of-days/243787/3 "2020-08-05T19:37:14Z")

</div>

Thanks for the reply @Badger. Unfortunately thats not a viable solution. I think best solution in this case would be to create another s3 bucket, ingest all logs there and copy over all log files into the existing buckets daily directories.

- push cloudflare logs to "new-logs-bucket"
- use s3-indexer.conf to point to that bucket
- copy over log files into my-bucket-of-logs/{{ current\_date }}
- create lifecycle policy to remove logs after N amount of days from "new-logs-bucket"

I am open to other suggestions as well.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 2, 2020, 7:37pm UTC](https://discuss.elastic.co/t/s3-logstash-setting-bucket-prefix-to-be-specific-range-of-days/243787/4 "2020-09-02T19:37:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.