# Unusual number of HEAD requests being made by S3 input plugin

**URL:** <https://discuss.elastic.co/t/unusual-number-of-head-requests-being-made-by-s3-input-plugin/23894>\
**Category:** Logstash\
**Created:** [June 18, 2015, 9:02am UTC](https://discuss.elastic.co/t/unusual-number-of-head-requests-being-made-by-s3-input-plugin/23894 "2015-06-18T09:02:24Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\
**Post date:** [June 18, 2015, 9:02am UTC](https://discuss.elastic.co/t/unusual-number-of-head-requests-being-made-by-s3-input-plugin/23894/1 "2015-06-18T09:02:24Z")

</div>

We have S3 access logs being collected in a bucket. We are using S3 input plugin to index these files into ELK.

After a couple of months usage we noticed unusual no of requests made  
to S3 (~1 Billion/Month) which costs $440, this is only the charge for  
the no of requests which is negligible for most of the use cases, and no  
one even bothers about this cost.

When I looked at the billing reports, there were around 950 Million HEAD reqeusts made to the bucket which has these logs.

S3 input plugin must be making all these requests (file watching?)

I am not sure if there is any need to do some kind of optimization on the plugin part.

I think the logs that people store in S3 don't change over time(my  
assumption), so if a file is indexed already, then there is no need to  
watch that.

From user perspective, the options I can think of, to avoid these requests are

1. Move the files to different location after the indexing is done, using backup\_to\_bucket option

2. Download the files to local drive using a cron job and use file input plugin to index to ES

3. Use daily prefixes, so that plugin watches only those files, log files are named with timestamps

4. Change the default interval to something higher if having some delay  
is fine, S3 access logs are hourly generated, so there is an hour delay  
anyway.

Any opinions and suggestions are welcome.

Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:37am UTC](https://discuss.elastic.co/t/unusual-number-of-head-requests-being-made-by-s3-input-plugin/23894/2 "2017-07-06T05:37:07Z")

</div>

Closed on July 6, 2017, 5:37am UTC