# Logstash Sincedb duplicate entries

**URL:** <https://discuss.elastic.co/t/logstash-sincedb-duplicate-entries/346187>\
**Category:** Logstash\
**Created:** [November 1, 2023, 10:14am UTC](https://discuss.elastic.co/t/logstash-sincedb-duplicate-entries/346187 "2023-11-01T10:14:36Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![justin\_sch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/justin_sch/32/121850_2.png) [@justin\_sch](https://discuss.elastic.co/u/justin_sch)\
**Post date:** [November 1, 2023, 10:14am UTC](https://discuss.elastic.co/t/logstash-sincedb-duplicate-entries/346187/1 "2023-11-01T10:14:36Z")

</div>

Hey,

I'm using the ELK-Stack to analyze a Log-File.  
Right now I clone the Log-File via SSH onto my local machine via a bash script every hour.  
The Logfile gets data appended every minute.  
This is my conf:

```auto
input {
  file {
    path => "Log.log"
    start_position => "end"
    sincedb_path => "etc/logstash/sincedb"
  }
}

filter {
  grok {
    match => {"Filtering"}
  }
  
  if "_grokparsefailure" in [tags] {
    drop { }
  }  
  date {
     match => ["timestamp", "yyyy-MM-dd HH:mm:ss"]
     target => "@timestamp"
  }  

  mutate {
    remove_field => ["message"] # Remove the 'message' and 'timestamp' fields if needed
  }
}

output {
  stdout { codec => rubydebug }
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "index1"
  }
}

```

The issue is that even though im using sincedb every hour when the log-file gets updated logstash starts reading it from the front again which leads to duplicates in my elasticsearch index.

Had a look at the sincedb file:

```auto
2063229 0 66305 34537335 1698822168.460246 Log1.log

2063229 0 66305 848366127 1698830975.2680178 Log1.log

```

As you can see it seems like the data of the logfile just gets added up in sincedb (?)

Does anyone of you have an idea what could be the issue?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 1, 2023, 5:46pm UTC](https://discuss.elastic.co/t/logstash-sincedb-duplicate-entries/346187/2 "2023-11-01T17:46:45Z")

</div>

> [@justin\_sch](#):
>
> Does anyone of you have an idea what could be the issue?

You are creating a new file each time you clone the log. The sincedb values ([here](https://github.com/logstash-plugins/logstash-input-file/blob/b2dd26249737c6dc426ef8efdda1c4227d9f6c5a/lib/filewatch/sincedb_record_serializer.rb#L34) and [here](https://github.com/logstash-plugins/logstash-input-file/blob/b2dd26249737c6dc426ef8efdda1c4227d9f6c5a/lib/filewatch/sincedb_record_serializer.rb#L34)) show a different minor device number. I guess you are working on Windows, so those are actually nFileIndexHigh/nFileIndexLow rather than major/minor device numbers.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 29, 2023, 5:46pm UTC](https://discuss.elastic.co/t/logstash-sincedb-duplicate-entries/346187/3 "2023-11-29T17:46:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.