# Logstash s3 input plugin with dynamic prefix **URL:** \ **Category:** Logstash\ **Created:** [April 3, 2018, 7:23pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-with-dynamic-prefix/126631 "2018-04-03T19:23:07Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![vikas-prabhakar](https://avatars.discourse-cdn.com/v4/letter/v/9dc877/32.png) [@vikas-prabhakar](https://discuss.elastic.co/u/vikas-prabhakar)\ **Post date:** [April 3, 2018, 7:23pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-with-dynamic-prefix/126631/1 "2018-04-03T19:23:07Z")
Hello, we have a special requirement where we have to read data from s3 bucket in which folders are dynamic(e.g s3bucket/2018/04/04) ---
**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\ **Post date:** [April 3, 2018, 11:01pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-with-dynamic-prefix/126631/2 "2018-04-03T23:01:14Z")
simply avoid setting a [`prefix`](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-s3.html#plugins-inputs-s3-prefix) and instead use an [`exclude_pattern`](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-s3.html#plugins-inputs-s3-exclude_pattern) to exclude files you don't want to include. ---
**Author:** ![vikas-prabhakar](https://avatars.discourse-cdn.com/v4/letter/v/9dc877/32.png) [@vikas-prabhakar](https://discuss.elastic.co/u/vikas-prabhakar)\ **Post date:** [April 4, 2018, 9:21am UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-with-dynamic-prefix/126631/3 "2018-04-04T09:21:21Z")
@yaauie thanks for the prompt reply , but scenario is something different , I have a service which creates folder like bucketname/MM/DD/YY so for each day there will be different folder , I want that prefix part in s3 input plugin can dynamically take something like this , prefix =\> "MM/DD/YY" , so everyday there will be new folder , where MM/DD/YY should be coming from current date . Thus using this i will processing current day file . But if there's some other work around do let me know . ---
**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\ **Post date:** [April 4, 2018, 10:47pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-with-dynamic-prefix/126631/4 "2018-04-04T22:47:30Z")
> [@vikas-prabhakar](#): > > I have a service which creates folder like bucketname/MM/DD/YY so for each day there will be different folder , I want that prefix part in s3 input plugin can dynamically take something like this , prefix =\> "MM/DD/YY" , so everyday there will be new folder , where MM/DD/YY should be coming from current date . > > Thus using this i will processing current day file . If you were to dynamically generate the prefix, things would get really weird around midnight. Interestingly enough, S3 buckets don't have "folders" -- they allow forward-slashes in filenames, and their web UI can present them in a folder-ish way, but the APIs used by Logstash and others treat all documents as if they were in a single flat bucket. When the Logstash S3 input points at a bucket, it "notices" _any_ new file that shows up based on creation timestamp, regardless of its path; if a `prefix` is specified, it will skip any files that don't start with that prefix, and if an `exclude_pattern` is specified, it will skip any file whose name matches the pattern. If you _only_ put logs that you want Logstash to read in this bucket, configure the plugin without a `prefix` or an `exclude_pattern`, and it will simply discover all files as they are added to the bucket. If the bucket also contains files that you _don't_ want Logstash to read, you have two options: - if the files that you _do_ want to read have a consistent, literal prefix, configure `logstash-input-s3` with a `prefix` directive - use the `exclude_pattern` to explicitly exclude files * * * > [@vikas-prabhakar](#): > > MM/DD/YY As an aside, if you do want to have the date be part of your filename/path, I would suggest an ISO-8601-compliant prefix, because it is also naturally _lexically_ ordered, which makes things a _lot_ easier in the long-run: ```auto YYYY/MM/DD ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 2, 2018, 10:47pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-with-dynamic-prefix/126631/5 "2018-05-02T22:47:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.