# Most recent CSV file from S3 gets read indefinitely

**URL:** https://discuss.elastic.co/t/most-recent-csv-file-from-s3-gets-read-indefinitely/191906
**Category:** Logstash
**Created:** [July 23, 2019, 8:09pm UTC](https://discuss.elastic.co/t/most-recent-csv-file-from-s3-gets-read-indefinitely/191906 "2019-07-23T20:09:17Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![resuni](https://avatars.discourse-cdn.com/v4/letter/r/ac8455/32.png) [@resuni](https://discuss.elastic.co/u/resuni)
#### Post date: [July 23, 2019, 8:09pm UTC](https://discuss.elastic.co/t/most-recent-csv-file-from-s3-gets-read-indefinitely/191906/1 "2019-07-23T20:09:17Z")

</div>

I'm trying to process CSV files stored in an S3 bucket using Logstash. Everything works fine until it gets to the last file, which it creates entries in Elasticsearch for endlessly.

The data is in daily time buckets, and each CSV file contains data for one day (grouped by various things). If I watch the document count in the Discover section of Kibana, a normal day shouldn't contain many more than 100,000 documents. The most recent day will continue to climb far into the millions before I stop Logstash.

As a troubleshooting step, I've removed the filter block from my config, and I still see the document count on that index go way higher than it should. This confirms it's not an issue with my filters, but perhaps an issue with the way I've configured my S3 plugin.

Here is that reduced config with some sensitive information censored:

```
input {

  s3 {
    type => "b"
    endpoint => "<s3-compatible storage URL>"
    access_key_id => "<redacted>"
    secret_access_key => "<redacted>"
    bucket => "b"
    sincedb_path => "/var/lib/logstash/plugins/inputs/s3/b.sincedb"
  }

}

output {

  if [type] == "b" {
    elasticsearch {
      hosts => ["localhost:9200"]
      index => "b"
    }
  }

}

```

I've tried with and without manually specifying the sincedb\_path, and it does the same thing.

I don't see what would cause something like this unless there's something fundamental I'm misunderstanding about the S3 plugin. Any thoughts?

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 23, 2019, 9:15pm UTC](https://discuss.elastic.co/t/most-recent-csv-file-from-s3-gets-read-indefinitely/191906/2 "2019-07-23T21:15:29Z")

</div>

You have not set watch\_for\_new\_files, and it defaults to true, so the s3 input sits in a loop, checking every file in the bucket to see if its last\_modified date is newer than the last time it saw that file. If it is, then it adds the file to the list of files it should read.

The way I read [the code](https://github.com/logstash-plugins/logstash-input-s3/blob/96f7e4a9a36f56681161ea176b37d4913f7fdef7/lib/logstash/inputs/s3.rb#L286), if it reads a file, it reads the whole file. It is not storing how far into the file it read, it is just re-reading the file, so if the last\_modified date changes, you get duplicate documents. There is an [issue](https://github.com/logstash-plugins/logstash-input-s3/issues/54) for this, but it hasn't been updated in years.

---

<div class="post-metadata">

### Author: ![resuni](https://avatars.discourse-cdn.com/v4/letter/r/ac8455/32.png) [@resuni](https://discuss.elastic.co/u/resuni)
#### Post date: [July 23, 2019, 10:12pm UTC](https://discuss.elastic.co/t/most-recent-csv-file-from-s3-gets-read-indefinitely/191906/3 "2019-07-23T22:12:58Z")

</div>

When a file is written to that S3 bucket, it's the entire file. This happens once a day. In my situation I shouldn't need it to keep track of how far into the file it reads.

I think I want watch\_for\_new\_files to be true, because I want it to be watching for that new file that comes in the next day.

I think my issue is more related to this bug: [https://github.com/logstash-plugins/logstash-input-s3/issues/172](https://github.com/logstash-plugins/logstash-input-s3/issues/172)

---

<div class="post-metadata">

### Author: ![resuni](https://avatars.discourse-cdn.com/v4/letter/r/ac8455/32.png) [@resuni](https://discuss.elastic.co/u/resuni)
#### Post date: [July 24, 2019, 4:47pm UTC](https://discuss.elastic.co/t/most-recent-csv-file-from-s3-gets-read-indefinitely/191906/4 "2019-07-24T16:47:54Z")

</div>

I made the code change that the guy in that issue mentioned, and it fixed my problem.

I don't know if I should mark this thread solved though, because modifying Logstash code doesn't seem like a good solution worth recommending to others.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 21, 2019, 4:58pm UTC](https://discuss.elastic.co/t/most-recent-csv-file-from-s3-gets-read-indefinitely/191906/5 "2019-08-21T16:58:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.