# Logstash produces duplicates
**URL:** https://discuss.elastic.co/t/logstash-produces-duplicates/35216
**Category:** Logstash
**Created:** [November 21, 2015, 12:07pm UTC](https://discuss.elastic.co/t/logstash-produces-duplicates/35216 "2015-11-21T12:07:48Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@vladmiller](https://discuss.elastic.co/u/vladmiller)
#### Post date: [November 21, 2015, 12:07pm UTC](https://discuss.elastic.co/t/logstash-produces-duplicates/35216/1 "2015-11-21T12:07:48Z")
Hello,
So the goal is to import existing mysql table which has about 2 million records in the ES index. However, ES index after a while has much more data.
I also try to generate unique sha1 fingerprint of each message and use it as document\_id to avoid, duplicates.
However, even trough original mysql table has 2m records, new ES index would have much more after a while.
What could be the problem and how is it possible to fix it?
Here is my config
> input {
> jdbc {
> jdbc\_driver\_library =\> "/app/bin/mysql-connector-java-5.1.37-bin.jar"
> jdbc\_driver\_class =\> "com.mysql.jdbc.Driver"
> jdbc\_connection\_string =\> "jdbc:mysql://testdatabase.xxxxxxxx.us-west-2.rds.amazonaws.com:3306/test"
> jdbc\_page\_size =\> 25000
> jdbc\_paging\_enabled =\> true
> statement =\> "SELECT \* FROM Table"
> }
> }
> filter {
> ruby {
> code =\> "
> require 'digest/sha1';
> event['fingerprint'] = Digest::SHA1.hexdigest(event.to\_json);
> "
> }
> }
> output {
> elasticsearch {
> hosts =\> ["host:80"]
> index =\> "fcblive"
> document\_type =\> "action"
> document\_id =\> "%{fingerprint}"
> }
> }
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [November 22, 2015, 9:13pm UTC](https://discuss.elastic.co/t/logstash-produces-duplicates/35216/2 "2015-11-22T21:13:30Z")
Are you ever restarting Logstash or does it produce duplicates even with a single Logstash execution?
Exactly what does an event look like? If Logstash adds the `@timestamp` field with the current time when a database record is read from the database the SHA-1 digest will be different every time a particular database record is processed.
---
### Author:  [@vladmiller](https://discuss.elastic.co/u/vladmiller)
#### Post date: [November 23, 2015, 8:38am UTC](https://discuss.elastic.co/t/logstash-produces-duplicates/35216/3 "2015-11-23T08:38:18Z")
It produces duplicates from a single logstash run.
I see your point, my fingerprint shouldn't work at all as expected because it will include `@timestamp` field. This however does not answer the question, why logstash would insert multiple duplicates.
I run commands with nohup, so basically if command fails it shouldn't be restarted.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 5:21am UTC](https://discuss.elastic.co/t/logstash-produces-duplicates/35216/4 "2017-07-06T05:21:45Z")