# How to avoid data duplication in elasticsearch when data send from logstash?

**URL:** <https://discuss.elastic.co/t/how-to-avoid-data-duplication-in-elasticsearch-when-data-send-from-logstash/149044>\
**Category:** Elasticsearch\
**Created:** [September 19, 2018, 2:55am UTC](https://discuss.elastic.co/t/how-to-avoid-data-duplication-in-elasticsearch-when-data-send-from-logstash/149044 "2018-09-19T02:55:39Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![nikhil.k](https://avatars.discourse-cdn.com/v4/letter/n/5daacb/32.png) [@nikhil.k](https://discuss.elastic.co/u/nikhil.k)\
**Post date:** [September 19, 2018, 2:55am UTC](https://discuss.elastic.co/t/how-to-avoid-data-duplication-in-elasticsearch-when-data-send-from-logstash/149044/1 "2018-09-19T02:55:39Z")

</div>

Hi all,

Logstash & Elasticsearch version - 5.4.3

Steps to produce the problem :-

1. Use the below logstash.conf file  
input {  
file {  
path =\> ["path of the file"]  
start\_position =\> "beginning"  
ignore\_older =\> 0  
}  
}

output {  
elasticsearch {  
hosts =\> "10.0.X.X"  
manage\_template =\> false  
index =\> "logstash-%{+YYYY.MM.dd}"  
}  
}

1. Start logstash

# bin/logstash -f config/logstash

1. Start elasticsearch

# systemctl start elasticsearch

1. During the transfer of logs from logstash to elasticsearch, restart the elasticsearch.

2. When the data transfer completes, the docs.counts value(1100) on elasticsearch is more than the no of lines of input file(1000).

Please give me the solution to avoid the data duplication in elasticsearch  
Regards  
Nikhil kapoor

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 19, 2018, 5:26am UTC](https://discuss.elastic.co/t/how-to-avoid-data-duplication-in-elasticsearch-when-data-send-from-logstash/149044/2 "2018-09-19T05:26:56Z")

</div>

Have a look at [this blog post](https://www.elastic.co/blog/logstash-lessons-handling-duplicates), which covers how to avoid duplicates.

---

<div class="post-metadata">

**Author:** ![nikhil.k](https://avatars.discourse-cdn.com/v4/letter/n/5daacb/32.png) [@nikhil.k](https://discuss.elastic.co/u/nikhil.k)\
**Post date:** [September 19, 2018, 6:28am UTC](https://discuss.elastic.co/t/how-to-avoid-data-duplication-in-elasticsearch-when-data-send-from-logstash/149044/3 "2018-09-19T06:28:41Z")

</div>

Hi @Christian_Dahlqvist,

I have gone through the given link and used the below logstash.conf file  
input{  
file{  
path =\> "path of log file"  
start\_position =\> "beginning"  
ignore\_older =\> 0  
}  
}

filter{  
grok{  
match =\> { "message" =\> "%{COMBINEDAPACHELOG}" }  
}  
}

output {  
elasticsearch {  
hosts =\> "10.x.x.x"  
manage\_template =\> false  
document\_id =\> "%{IPORHOST:clientip}"  
index =\> "logstash-%{+YYYY.MM.dd}"  
}  
}

In normal scenario:-  
When logs are transferred from logstash to elasticsearch, it is observed that the no of input lines of log file are "5" and the docs.count value is "1". Below is the output:-

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/8/18d27d05ee416f22b8dcd5ab36ddac49a09972cc.png)

Can you just help me how to use document\_id to avoid duplication of data?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 19, 2018, 6:58am UTC](https://discuss.elastic.co/t/how-to-avoid-data-duplication-in-elasticsearch-when-data-send-from-logstash/149044/4 "2018-09-19T06:58:09Z")

</div>

Did you read the blog I linked to? Use the fingerprint filter on the message, e.g. with a MD5 or SHA1 hash (MURMUR3 generally has too high collision risk) and then use this fingerprint as document id.

---

<div class="post-metadata">

**Author:** ![nikhil.k](https://avatars.discourse-cdn.com/v4/letter/n/5daacb/32.png) [@nikhil.k](https://discuss.elastic.co/u/nikhil.k)\
**Post date:** [September 19, 2018, 10:49am UTC](https://discuss.elastic.co/t/how-to-avoid-data-duplication-in-elasticsearch-when-data-send-from-logstash/149044/5 "2018-09-19T10:49:43Z")

</div>

Thanks @Christian_Dahlqvist for the replies.  
Fingerprint filter solved my problem.

Regards  
Nikhil Kapoor

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 17, 2018, 10:49am UTC](https://discuss.elastic.co/t/how-to-avoid-data-duplication-in-elasticsearch-when-data-send-from-logstash/149044/6 "2018-10-17T10:49:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.