# S3 input plugin - pull the files from only specific folder

**URL:** <https://discuss.elastic.co/t/s3-input-plugin-pull-the-files-from-only-specific-folder/131331>\
**Category:** Logstash\
**Created:** [May 10, 2018, 4:22pm UTC](https://discuss.elastic.co/t/s3-input-plugin-pull-the-files-from-only-specific-folder/131331 "2018-05-10T16:22:26Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![mruthyu](https://avatars.discourse-cdn.com/v4/letter/m/bb73d2/32.png) [@mruthyu](https://discuss.elastic.co/u/mruthyu)\
**Post date:** [May 10, 2018, 4:22pm UTC](https://discuss.elastic.co/t/s3-input-plugin-pull-the-files-from-only-specific-folder/131331/1 "2018-05-10T16:22:26Z")

</div>

I want to index the files from S3 to elasticsearch. I am not able to see the option of specifying the folder in the bucket for s3 input plugin. Is there any option to mention only the specific path instead of pulling the files from the complete bucket?

---

<div class="post-metadata">

**Author:** ![mruthyu](https://avatars.discourse-cdn.com/v4/letter/m/bb73d2/32.png) [@mruthyu](https://discuss.elastic.co/u/mruthyu)\
**Post date:** [May 11, 2018, 10:31am UTC](https://discuss.elastic.co/t/s3-input-plugin-pull-the-files-from-only-specific-folder/131331/2 "2018-05-11T10:31:52Z")

</div>

Is it possible to mention the specific path along with the S3 bucket,as we don't want to pull all the data from all folders from the bucket.

---

<div class="post-metadata">

**Author:** ![bjohnson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bjohnson/32/31126_2.png) [@bjohnson](https://discuss.elastic.co/u/bjohnson)\
**Post date:** [May 11, 2018, 2:51pm UTC](https://discuss.elastic.co/t/s3-input-plugin-pull-the-files-from-only-specific-folder/131331/3 "2018-05-11T14:51:15Z")

</div>

You can use the [prefix](https://www.elastic.co/guide/en/logstash-versioned-plugins/versioned_plugin_docs/v3.3.4-plugins-inputs-s3.html#v3.3.4-plugins-inputs-s3-prefix) property for this:

```
input {
  s3 {
    # ...
    prefix => "/path/to/files/you/want/"
  }
}
```

---

<div class="post-metadata">

**Author:** ![mruthyu](https://avatars.discourse-cdn.com/v4/letter/m/bb73d2/32.png) [@mruthyu](https://discuss.elastic.co/u/mruthyu)\
**Post date:** [May 14, 2018, 8:45am UTC](https://discuss.elastic.co/t/s3-input-plugin-pull-the-files-from-only-specific-folder/131331/4 "2018-05-14T08:45:03Z")

</div>

Thanks. Tried the prefix option and none of the data from the files ( within folders ) are getting pulled. But when I comment out this statement and only the bucket property is present then I could see it is working. So still checking how to make it work for a specific folder.

---

<div class="post-metadata">

**Author:** ![mruthyu](https://avatars.discourse-cdn.com/v4/letter/m/bb73d2/32.png) [@mruthyu](https://discuss.elastic.co/u/mruthyu)\
**Post date:** [May 14, 2018, 8:56am UTC](https://discuss.elastic.co/t/s3-input-plugin-pull-the-files-from-only-specific-folder/131331/5 "2018-05-14T08:56:34Z")

</div>

Looks like the prefix parameter is for the files within the buckets and not for the folder.

**If specified, the prefix of filenames in the bucket must match (not a regexp)**

I want something like this.  
/bucket name/datewise folder name/all files needs to be read

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 11, 2018, 8:56am UTC](https://discuss.elastic.co/t/s3-input-plugin-pull-the-files-from-only-specific-folder/131331/6 "2018-06-11T08:56:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.