# Duplicates entries when using S3 input

**URL:** <https://discuss.elastic.co/t/duplicates-entries-when-using-s3-input/205932>\
**Category:** Logstash\
**Tags:** docker\
**Created:** [October 30, 2019, 6:25pm UTC](https://discuss.elastic.co/t/duplicates-entries-when-using-s3-input/205932 "2019-10-30T18:25:46Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![cascoalessio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cascoalessio/32/56880_2.png) [@cascoalessio](https://discuss.elastic.co/u/cascoalessio)\
**Post date:** [October 30, 2019, 6:25pm UTC](https://discuss.elastic.co/t/duplicates-entries-when-using-s3-input/205932/1 "2019-10-30T18:25:46Z")

</div>

Hello Guys

I just deployed on k8s a logstash deployment that reads logs from S3 and pushes them to Elasticsearch.

I noticed that the sincedb file when using the S3 input has this content:  
`2019-10-30 16:14:08 UTC`

So the granularity goes down only to the second where it left reading.

What I've noticed is that if you have multiple log lines within the second and logstash restarts, you end up having either duplicates or having missing data due to the fact that it has to either move to the next second or read from the beginning of it.  
I've tested a bit and It seems that it starts reading the whole second written in the sincedb so in my tests I got duplicates.

Is there anything I can do in order to fix this problem?

Thanks  
Alessio

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 27, 2019, 6:25pm UTC](https://discuss.elastic.co/t/duplicates-entries-when-using-s3-input/205932/2 "2019-11-27T18:25:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.