# Duplicate log entries **URL:** \ **Category:** Elasticsearch\ **Created:** [December 22, 2020, 2:41pm UTC](https://discuss.elastic.co/t/duplicate-log-entries/259394 "2020-12-22T14:41:40Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![leandro.borges](https://avatars.discourse-cdn.com/v4/letter/l/ce73a5/32.png) [@leandro.borges](https://discuss.elastic.co/u/leandro.borges)\ **Post date:** [December 22, 2020, 2:41pm UTC](https://discuss.elastic.co/t/duplicate-log-entries/259394/1 "2020-12-22T14:41:40Z")
Hi, We are facing some issues with elasticsearch. We are having lots of duplicate log entries like below. Inside logstash.yml there is only a file in path.config that we use: path.config: "/etc/logstash/pipeline.global.conf" ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/3/2330c2b88b14215eb8f32c4e98d9a9a108d56380.png) Does someone has any idea on how to fix this issue ? Thank you ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 22, 2020, 2:55pm UTC](https://discuss.elastic.co/t/duplicate-log-entries/259394/2 "2020-12-22T14:55:21Z")
Where does the duplicated data reside? Is it by any chance a shared drive? How is it being indexed into Elasticsearch? ---
**Author:** ![leandro.borges](https://avatars.discourse-cdn.com/v4/letter/l/ce73a5/32.png) [@leandro.borges](https://discuss.elastic.co/u/leandro.borges)\ **Post date:** [December 22, 2020, 3:22pm UTC](https://discuss.elastic.co/t/duplicate-log-entries/259394/3 "2020-12-22T15:22:03Z")
Hi Christian Thank you for your reply. The origin of logs are inside shared drives but they are not duplicated. We use filebeat and logstash with the following pipeline configuration: if [client] == "iis" { if [indexname] { elasticsearch { hosts =\> ["[https://xxxxxx:9200](https://xxxxxx:9200)" ] index=\> "rq-%{[client]}-%{[indexname]}-%{+YYYY.MM.dd}" ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 22, 2020, 3:41pm UTC](https://discuss.elastic.co/t/duplicate-log-entries/259394/4 "2020-12-22T15:41:02Z")
Logstash and Filebeat can have issues reading from network drives, [which is not recommended](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-network-volumes.html). This may very well be why you are seeing duplicates. There could also be other reasons, e.g. issues with your Logstash config, but it is hard to tell without more details around config and how frequent the duplication issue is. ---
**Author:** ![leandro.borges](https://avatars.discourse-cdn.com/v4/letter/l/ce73a5/32.png) [@leandro.borges](https://discuss.elastic.co/u/leandro.borges)\ **Post date:** [December 22, 2020, 4:03pm UTC](https://discuss.elastic.co/t/duplicate-log-entries/259394/5 "2020-12-22T16:03:40Z")
Sorry, the drive is shared but filebeat takes it from the drive physically reside inside the same machine, not from a network. ---
**Author:** ![leandro.borges](https://avatars.discourse-cdn.com/v4/letter/l/ce73a5/32.png) [@leandro.borges](https://discuss.elastic.co/u/leandro.borges)\ **Post date:** [December 22, 2020, 4:57pm UTC](https://discuss.elastic.co/t/duplicate-log-entries/259394/6 "2020-12-22T16:57:30Z")
Can I send you my pipeline config file? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 22, 2020, 5:07pm UTC](https://discuss.elastic.co/t/duplicate-log-entries/259394/7 "2020-12-22T17:07:51Z")
Hmmm, the two documents are not necessarily duplicates as at least `log.offset` differs. There may not be anything wrong at all... ---
**Author:** ![leandro.borges](https://avatars.discourse-cdn.com/v4/letter/l/ce73a5/32.png) [@leandro.borges](https://discuss.elastic.co/u/leandro.borges)\ **Post date:** [December 22, 2020, 7:00pm UTC](https://discuss.elastic.co/t/duplicate-log-entries/259394/8 "2020-12-22T19:00:20Z")
In that case you are right. But in another example, we checked the log file and there wasn´t duplicated lines. In this case, only the \_id and the ingest time are different: ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/a/6ad99bb1dc922157d7f2cde78f9710ae6bb2220c.png) ---
**Author:** ![gerilya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gerilya/32/22317_2.png) [@gerilya](https://discuss.elastic.co/u/gerilya)\ **Post date:** [December 23, 2020, 7:47am UTC](https://discuss.elastic.co/t/duplicate-log-entries/259394/9 "2020-12-23T07:47:51Z")
If you are sending documents to Elasticsearch in bulks and do not specify custom document\_id, you can not guarantee exactly once delivery. When ES cluster is busy it might "reject" indexing requests (so-called back pressure). In case of bulk requests, a coordinating node will split the request into smaller "sub-requests" (1 per shard) and will send all requests in parallel. In such scenario, some sub-requests can complete successfully while others might fail resulting in partial rejection. Your logstash might retry the whole bulk though. If you provide your own document\_id, it should fix the duplicate issue but it will affect you indexing performance. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 23, 2020, 8:24am UTC](https://discuss.elastic.co/t/duplicate-log-entries/259394/10 "2020-12-23T08:24:48Z")
That is a significant difference in ingest time, so suggests it is not due to retries when indexing into Elasticsearch. I would recommend searching for other log entries from that file around that time and check if those also are duplicated. If it is the entire file it would seem like something happened at the file system level, causing the file to be reprocessed, but it is hard to tell without being able to investigate the data directly. ---
**Author:** ![leandro.borges](https://avatars.discourse-cdn.com/v4/letter/l/ce73a5/32.png) [@leandro.borges](https://discuss.elastic.co/u/leandro.borges)\ **Post date:** [December 23, 2020, 1:18pm UTC](https://discuss.elastic.co/t/duplicate-log-entries/259394/11 "2020-12-23T13:18:27Z")
Hi Thank you for your answer. I checked the file itself and there is no duplicate lines. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 23, 2020, 1:19pm UTC](https://discuss.elastic.co/t/duplicate-log-entries/259394/12 "2020-12-23T13:19:54Z")
Are you saying that this is the only document that has been duplicated in Elasticsearch from that file? ---
**Author:** ![leandro.borges](https://avatars.discourse-cdn.com/v4/letter/l/ce73a5/32.png) [@leandro.borges](https://discuss.elastic.co/u/leandro.borges)\ **Post date:** [December 23, 2020, 1:34pm UTC](https://discuss.elastic.co/t/duplicate-log-entries/259394/13 "2020-12-23T13:34:28Z")
No, we have for example 6 identical log entries with different ingest times. ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/8/f876c7c2aaf9cd4b90f46782be94d83754f46aae.png) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 23, 2020, 1:36pm UTC](https://discuss.elastic.co/t/duplicate-log-entries/259394/14 "2020-12-23T13:36:08Z")
Yes, but is it just one entry that has been duplicated 6 times or have ALL entries (at least the initial ones) been duplicated as well? ---
**Author:** ![leandro.borges](https://avatars.discourse-cdn.com/v4/letter/l/ce73a5/32.png) [@leandro.borges](https://discuss.elastic.co/u/leandro.borges)\ **Post date:** [December 23, 2020, 1:43pm UTC](https://discuss.elastic.co/t/duplicate-log-entries/259394/15 "2020-12-23T13:43:07Z")
It is one entry from the log file that has been duplicated 6 times in elasticsearch. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 23, 2020, 1:52pm UTC](https://discuss.elastic.co/t/duplicate-log-entries/259394/16 "2020-12-23T13:52:04Z")
If that is one of only a few duplicated entries from the file I am not sure where the issue is. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [December 23, 2020, 3:26pm UTC](https://discuss.elastic.co/t/duplicate-log-entries/259394/17 "2020-12-23T15:26:38Z")
Can you share your full logstash pipeline? ---
**Author:** ![leandro.borges](https://avatars.discourse-cdn.com/v4/letter/l/ce73a5/32.png) [@leandro.borges](https://discuss.elastic.co/u/leandro.borges)\ **Post date:** [December 23, 2020, 3:52pm UTC](https://discuss.elastic.co/t/duplicate-log-entries/259394/18 "2020-12-23T15:52:32Z")
Ola, vide abaixo Obrigado ``` input { beats { port => 5044 } } filter { if [fields][client] { mutate { add_field => { "client" => "%{[fields][client]}" } remove_field => ["[fields][client]"] } if [fields][indexname] { mutate { add_field => { "indexname" => "%{[fields][indexname]}" } remove_field => ["[fields][indexname]"] } } } if [client] == "wso2" { ######### Extracting message elements ######### grok { patterns_dir => ["/etc/logstash/patterns/extra_patterns"] #wso2carbon-log match => { "message" => "TID:%{SPACE}\[(%{INT:tenant-id})?\]%{SPACE}\[(%{WORD:server-type})?\]%{SPACE}\[%{TIMESTAMP_ISO8601:date-time}\]%{SPACE}%{LOGLEVEL:level}%{SPACE}{%{JAVACLASS:logger-name}}%{SPACE}-%{SPACE}%{LOGMESSAGE:log-message}((\r?\n)%{STACKTRACE:stacktrace})?" } #wso2errors-log, wso2-service-log match => { "message" => "%{TIMESTAMP_ISO8601:date-time}%{SPACE}\[%{NOTSPACE:server-type}\]%{SPACE}\[%{GREEDYDATA:logger-name}\]%{SPACE}%{LOGLEVEL:level}%{SPACE}%{NOTSPACE:component}%{SPACE}(%{LOGMESSAGE:log-message})?((\r?\n)%{STACKTRACE:stacktrace})?" } #wso2httpaccessmanagementconsole-log match => { "message" => "%{IP:ip-address}%{SPACE}-%{SPACE}-%{SPACE}\[%{HTTPDATE:date-time}\]%{SPACE}%{QS:method}%{SPACE}%{INT:status}%{SPACE}%{NOTSPACE:time-taken}%{SPACE}%{QS:uri-stem}%{SPACE}%{QS:user-agent}" } #wso2trace-log match => { "message" => "%{TIME:time}%{SPACE}\[-\]%{SPACE}\[%{NOTSPACE:logger-name}\]%{SPACE}%{LOGLEVEL:level}%{SPACE}%{NOTSPACE:component}%{SPACE}%{LOGMESSAGE:log-message}" } #ws02patches-log match => { "message" => "\[%{TIMESTAMP_ISO8601:date-time}\]%{SPACE}%{LOGLEVEL:level}%{SPACE}\{%{JAVACLASS:logger-name}\}%{SPACE}-%{SPACE}(%{LOGMESSAGE:log-message})?" } #wso2audit-log match => { "message" => "\[%{TIMESTAMP_ISO8601:date-time}\]%{SPACE}%{LOGLEVEL:level}%{SPACE}-%{SPACE}(%{LOGMESSAGE:log-message})?" } #wso2atomiktransaction-log match => { "message" => "%{CAPLOGLEVEL:level}%{SPACE}%{LOGMESSAGE:log-message}((\r?\n)%{STACKTRACE:stacktrace})?" } } ######### Updating timestamp field ######### if [type] == "wso2trace_log" { grok { match => { "@timestamp" => "%{DATE:date}[T]%{TIME}" } add_field => { "date-time" => "%{date}:%{time}" } } } date { match => ["date-time", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss", "dd/MMM/yyy:HH:mm:ss Z", "yy-MM-dd:HH:mm:ss,SSS"] target => "@timestamp" } ######### Removing unused fields ######### if [type] == "wso2httpaccessmanagementconsole_log" { mutate { remove_field => ["message"] } } mutate { update => { "message" => "%{log-message}" } remove_field => ["date", "time", "date-time", "log-message"] } } if [event][module] == "iis" { if [client] { mutate { update => { "client" => "iis" } } } else { mutate { add_field => { "client" => "iis" } } } mutate { lowercase => ["indexname"] } } } output { if [client] { if [client] == "iis" { if [indexname] { elasticsearch { hosts => [" "] index=> "rq-%{[client]}-%{[indexname]}-%{+YYYY.MM.dd}" cacert => " " user=> " " password=> " " } } else { elasticsearch { hosts => [" "] index=> "rq-%{[client]}-%{+YYYY.MM.dd}" cacert => " " user=> " " password=> " " } } } if [client] == "winlog" { if [indexname] { elasticsearch { hosts => [" "] index=> "rq-%{[client]}-%{[indexname]}-%{+YYYY.MM}" cacert => " " user=> " " password=> " " } } } else { if [indexname] { elasticsearch { hosts => [" "] index=> "rq-%{[client]}-%{[indexname]}-%{+YYYY.MM.dd}" cacert => " " user=> " " password=> " " } } else { elasticsearch { hosts => [" "] index=> "rq-%{[client]}-%{+YYYY.MM.dd}" cacert => " " user=> " " password=> " " } } } } else { elasticsearch { hosts => [" "] index=> "rq-%{+YYYY.MM.dd}" cacert => " " user=> " " password=> " " } } } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 20, 2021, 3:52pm UTC](https://discuss.elastic.co/t/duplicate-log-entries/259394/19 "2021-01-20T15:52:37Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.