# Duplicate record exatly twice **URL:** \ **Category:** Logstash\ **Created:** [March 22, 2018, 4:37pm UTC](https://discuss.elastic.co/t/duplicate-record-exatly-twice/125235 "2018-03-22T16:37:02Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![david.derosa](https://avatars.discourse-cdn.com/v4/letter/d/f08c70/32.png) [@david.derosa](https://discuss.elastic.co/u/david.derosa)\ **Post date:** [March 22, 2018, 4:37pm UTC](https://discuss.elastic.co/t/duplicate-record-exatly-twice/125235/1 "2018-03-22T16:37:02Z")
Hi all, i use elastisearch 6.2.2 in a really basic test environment (no cluster, single node). I've just realized that **all** records are duplicated exactly twice. The two records have different \_id but all other fields are the same. Only a short example: record 1 @timestamp March 22nd 2018, 17:17:20.700 @version 1 **\_id HEGATmIBZYl-EB49OMRc** \_index logstash-2018.03.22 message Connection lost after 0 seconds record 2 @timestamp March 22nd 2018, 17:17:20.700 @version 1 **\_id JUGATmIBZYl-EB49OsRI** \_index logstash-2018.03.22 message Connection lost after 0 seconds I configured logstash in debug mode but i didn't see any duplicate data. What could create this situation? thank you all. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 22, 2018, 7:09pm UTC](https://discuss.elastic.co/t/duplicate-record-exatly-twice/125235/2 "2018-03-22T19:09:39Z")
May be you ran twice logstash? ---
**Author:** ![david.derosa](https://avatars.discourse-cdn.com/v4/letter/d/f08c70/32.png) [@david.derosa](https://discuss.elastic.co/u/david.derosa)\ **Post date:** [March 22, 2018, 7:18pm UTC](https://discuss.elastic.co/t/duplicate-record-exatly-twice/125235/3 "2018-03-22T19:18:49Z")
There is only one logstash instance (one pid). what can i do to discover where the problem came from? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 22, 2018, 8:04pm UTC](https://discuss.elastic.co/t/duplicate-record-exatly-twice/125235/4 "2018-03-22T20:04:41Z")
Elasticsearch does not duplicate documents. It can only happen if logstash is sending twice the documents. May be a bug but I think you should see some traces in logstash logs or something you did wrong. May be share your logstash configuration file? I moved your question to #logstash ---
**Author:** ![david.derosa](https://avatars.discourse-cdn.com/v4/letter/d/f08c70/32.png) [@david.derosa](https://discuss.elastic.co/u/david.derosa)\ **Post date:** [March 23, 2018, 4:02pm UTC](https://discuss.elastic.co/t/duplicate-record-exatly-twice/125235/5 "2018-03-23T16:02:43Z")
I tried to add a second elasticsearch ouput in logstash and the data is not duplicate in this one but the data is always double in the first elasticsearch. This is my logstash.conf: ``` input { udp { host => "127.0.0.1" port => 10514 codec => "json" type => "rsyslog" } } # This is an empty filter block. You can later add other filters here to further process # your log lines #https://gist.github.com/mesimeris/bf6cd912d11b674c4a2b filter { if [sysloghost] == "192.168.0.1" { grok { patterns_dir => "/etc/logstash/grok/mikrotik.pattern" match => { "message" => "%{MIKROTIKFIREWALL}" } } } } # This output block will send all events of type "rsyslog" to Elasticsearch at the configured # host and port into daily indices of the pattern, "rsyslog-YYYY.MM.DD" output { if "_grokparsefailure" in [tags] { # write events that didn't match to a file file { "path" => "/tmp/grok_failures.txt" } } else { if [type] == "rsyslog" { elasticsearch { hosts => ["127.0.0.1:9200"] } elasticsearch { hosts => ["192.168.0.122:9200"] } stdout { codec => rubydebug } } } } ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 23, 2018, 4:13pm UTC](https://discuss.elastic.co/t/duplicate-record-exatly-twice/125235/6 "2018-03-23T16:13:14Z")
Logstash concatenates config files it finds in its directory. Do you by any chance have any other config files that specify the same Elasticsearch output? ---
**Author:** ![david.derosa](https://avatars.discourse-cdn.com/v4/letter/d/f08c70/32.png) [@david.derosa](https://discuss.elastic.co/u/david.derosa)\ **Post date:** [March 23, 2018, 4:27pm UTC](https://discuss.elastic.co/t/duplicate-record-exatly-twice/125235/7 "2018-03-23T16:27:10Z")
Yes, i have other two logstash configs with the same output but different inputs and filters. I add the second elastichsearch to second logstash config. The data in always not duplicate on the second and duplicate on the first. ☹ ``` input { # this is the actual live log file to monitor #file { # path => ["/home/cowrie/cowrie-git/log/cowrie.json"] # codec => json # type => "cowrie" #} # this is to send old logs to for reprocessing #tcp { # port => 3333 # type => "cowrie" #} beats { port => 5044 type => "cowrie" } } filter { if [type] == "cowrie" { json { source => message } date { match => ["timestamp", "ISO8601"] } mutate { convert => { "dst_port" => "integer" } convert => { "src_port" => "integer" } } if [src_ip] { mutate { add_field => { "src_host" => "%{src_ip}" } } dns { reverse => ["src_host"] nameserver => ["192.168.0.1"] action => "replace" hit_cache_size => 4096 hit_cache_ttl => 900 failed_cache_size => 512 failed_cache_ttl => 900 } geoip { source => "src_ip" target => "geoip" #database => "/opt/logstash/vendor/geoip/GeoLite2-City.dat" database => "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb" } } mutate { remove_tag => ["beats_input_codec_plain_applied"] # cut useless fields added by filebeat, if you use it of course remove_field => ["source", "offset", "input_type"] } } } output { if [type] == "cowrie" { elasticsearch { hosts => ["localhost:9200"] } elasticsearch { hosts => ["192.168.0.122:9200"] } # file { # path => "/tmp/cowrie-logstash.log" # codec => json # } # stdout { # codec => rubydebug # } } } ``` ---
**Author:** ![david.derosa](https://avatars.discourse-cdn.com/v4/letter/d/f08c70/32.png) [@david.derosa](https://discuss.elastic.co/u/david.derosa)\ **Post date:** [March 23, 2018, 7:06pm UTC](https://discuss.elastic.co/t/duplicate-record-exatly-twice/125235/8 "2018-03-23T19:06:00Z")
Thank you all for your support and tips. I found the configuration error in my last logstash conf file. The line of "if type" was commented in the output. It's a stupid error but not so easy to see. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 20, 2018, 7:06pm UTC](https://discuss.elastic.co/t/duplicate-record-exatly-twice/125235/9 "2018-04-20T19:06:07Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.