# Dupilcate message on elasticsearch **URL:** \ **Category:** Logstash\ **Created:** [November 10, 2017, 12:10pm UTC](https://discuss.elastic.co/t/dupilcate-message-on-elasticsearch/107182 "2017-11-10T12:10:42Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![pope843](https://avatars.discourse-cdn.com/v4/letter/p/f4b2a3/32.png) [@pope843](https://discuss.elastic.co/u/pope843)\ **Post date:** [November 10, 2017, 12:10pm UTC](https://discuss.elastic.co/t/dupilcate-message-on-elasticsearch/107182/1 "2017-11-10T12:10:42Z")
Hello Guys, Need help regarding duplicate messages on my elasticsearch. I have below config. As you can see i'm using fingerprint plugin to fix duplicates but to no avail. input { beats { port =\> 5044 ssl =\> true ssl\_certificate =\> "/etc/pki/tls/certs/logstash-forwarder.crt" ssl\_key =\> "/etc/pki/tls/private/logstash-forwarder.key" } } filter { if [type] == "syslog" { grok { match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: %{GREEDYDATA:syslog\_message}" } add\_field =\> ["received\_at", "%{@timestamp}"] add\_field =\> ["received\_from", "%{host}"] } syslog\_pri { } date { match =\> ["syslog\_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss", "HH:mm:ss.SSSZZZ"] } } else if [type] == "applog" { grok { match =\> {"message" =\> "%{DATA:timestampko} .\*"} add\_field =\> ["received\_at", "%{@timestamp}"] add\_field =\> ["received\_from", "%{host}"] } if "UTC " in [message] { if "23:59:5" not in [message] { ruby { code =\> " event.set('dateko', Time.now.strftime('%Y-%m-%d')) " } mutate { add\_field =\> { "akkatimestamp" =\> "%{dateko}.%{timestampko}" } remove\_field =\> ["timestampko", "newtimestamp"] } date { match =\> ["akkatimestamp" , "YYYY-MM-dd.HH:mm:ss.SSSZZZ", "yyyy-MM-dd.HH:mm:ss.SSSZZZ"] timezone =\> "UTC" target =\> "@timestamp" } } else { ruby { code =\> " event.set('datemo', DateTime.yesterday.strftime('%Y-%m-%d')) " } mutate { add\_field =\> { "akkatimestamp" =\> "%{datemo}.%{timestampko}" } remove\_field =\> ["timestampko", "newtimestamp"] } date { match =\> ["akkatimestamp" , "YYYY-MM-dd.HH:mm:ss.SSSZZZ", "yyyy-MM-dd.HH:mm:ss.SSSZZZ"] timezone =\> "UTC" target =\> "@timestamp" } } } else { grok { match =\> {"message" =\> "[%{DATA:timestamp}] .\*"} } ruby { code =\> " event.set('date', Time.now.strftime('%L')) " } ruby { code =\> " event.set('zone', Time.now.strftime('%Z')) " } mutate { add\_field =\> { "newtimestamp" =\> "%{timestamp}.%{date}%{zone}" } remove\_field =\> ["date", "timestamp", "zone"] } date { match =\> ["newtimestamp", "YYYY/MM/dd HH:mm:ss.SSSZZZ"] } } } if "DEBUG" not in [message] and "WARN" not in [message] and "INFO" not in [message] and "ERROR" not in [message] { drop { } } fingerprint { source =\> "message" target =\> "%{[@metadata][fingerprint]}" method =\> "MURMUR3" } } output { if [host] == "ip-x" or [host] == "ip-x" { if [type] == "applog" { elasticsearch { hosts =\> ["[https://search-em-escluster-rwzyzlh63zokr7bqssxjgerfbe.eu-west-1.es.amazonaws.com:443](https://search-em-escluster-rwzyzlh63zokr7bqssxjgerfbe.eu-west-1.es.amazonaws.com:443)"] sniffing =\> false manage\_template =\> false index =\> "applog-%{+YYYY.MM.dd}" document\_type =\> "%{[@metadata][type]}" document\_id =\> "%{[@metadata][fingerprint]}" } } else if [type] == "syslog" { elasticsearch { hosts =\> ["[https://search-em-escluster-rwzyzlh63zokr7bqssxjgerfbe.eu-west-1.es.amazonaws.com:443](https://search-em-escluster-rwzyzlh63zokr7bqssxjgerfbe.eu-west-1.es.amazonaws.com:443)"] sniffing =\> false manage\_template =\> false index =\> "syslog-%{+YYYY.MM.dd}" document\_type =\> "%{[@metadata][type]}" } } else { file { path =\> "/var/log/logstash/unknown\_messages\_hlr.log" } } } else { file { path =\> "/var/log/logstash/unknown\_messages.log" } } } ---
**Author:** ![pope843](https://avatars.discourse-cdn.com/v4/letter/p/f4b2a3/32.png) [@pope843](https://discuss.elastic.co/u/pope843)\ **Post date:** [November 11, 2017, 4:04am UTC](https://discuss.elastic.co/t/dupilcate-message-on-elasticsearch/107182/2 "2017-11-11T04:04:04Z")
Hello Guys i've also tried solution like below config. But i still encountered duplicate message on my ES. the only difference is the "\_id" field. Any thing you've adjusted other that above solution? fingerprint { source =\> "message" target =\> "%{[@metadata][fingerprint]}" method =\> "MURMUR3" } } output { if [host] == "ip-10-100-x" or [host] == "ip-10-100-x" { if [type] == "applog" { elasticsearch { hosts =\> ["host:443"] sniffing =\> false manage\_template =\> false index =\> "hlr-applog-%{+YYYY.MM.dd}" document\_type =\> "%{[@metadata][type]}" document\_id =\> "%{[@metadata][fingerprint]}" } } else if [type] == "syslog" { elasticsearch { hosts =\> ["host:443"] sniffing =\> false manage\_template =\> false index =\> "hlr-syslog-%{+YYYY.MM.dd}" document\_type =\> "%{[@metadata][type]}" document\_id =\> "%{[@metadata][fingerprint]}" } } else { file { path =\> "/var/log/logstash/unknown\_messages\_x.log" } } } ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 11, 2017, 11:22am UTC](https://discuss.elastic.co/t/dupilcate-message-on-elasticsearch/107182/3 "2017-11-11T11:22:16Z")
Can you show us two events in Elasticsearch that are duplicates? ---
**Author:** ![pope843](https://avatars.discourse-cdn.com/v4/letter/p/f4b2a3/32.png) [@pope843](https://discuss.elastic.co/u/pope843)\ **Post date:** [November 17, 2017, 6:10am UTC](https://discuss.elastic.co/t/dupilcate-message-on-elasticsearch/107182/4 "2017-11-17T06:10:00Z")
Hello it was resolved after i've removed document\_id =\> "%{[@metadata][fingerprint]}". Just one question. I've created an index template on ES and i want to use it on my rotated index-%{+YYYY.MM.dd}" every creation. how can i set this on logstash output? ---
**Author:** ![pope843](https://avatars.discourse-cdn.com/v4/letter/p/f4b2a3/32.png) [@pope843](https://discuss.elastic.co/u/pope843)\ **Post date:** [November 17, 2017, 9:02am UTC](https://discuss.elastic.co/t/dupilcate-message-on-elasticsearch/107182/5 "2017-11-17T09:02:50Z")
i have this error [2017-11-17T08:05:35,713][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=\>"/home/ubuntu/ELK/my-akkatemplate.json"} [2017-11-17T08:05:35,717][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage\_template=\>{"my\_akkatemplate"=\>{"order"=\>0, "template"=\>"[_applog_, em-hlr-applog-\*]", "settings"=\>{"index"=\>{"analysis"=\>{"index"=\>{"number\_of\_shards"=\>"3", "number\_of\_replicas"=\>"1"}, "analyzer"=\>{"domain\_name\_analyzer"=\>{"filter"=\>"lowercase", "type"=\>"custom", "tokenizer"=\>"domain\_name\_tokenizer"}}, "tokenizer"=\>{"domain\_name\_tokenizer"=\>{"reverse"=\>"true", "type"=\>"PathHierarchy", "delimiter"=\>"."}}}}}, "mappings"=\>{"applog"=\>{"properties"=\>{"message"=\>{"type"=\>"text", "fields"=\>{"ws"=\>{"analyzer"=\>"whitespace", "type"=\>"text"}}}}}}, "aliases"=\>{}}}} [2017-11-17T08:05:35,721][INFO][logstash.outputs.elasticsearch] Installing elasticsearch template to \_template/logstash [2017-11-17T08:05:35,725][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=\>"Got response code '400' contacting Elasticsearch at URL 'privte-host:443/\_template/logstash'", :class=\>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError", :backtrace=\>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/http\_client/manticore\_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:269:in`perform\_request\_to\_url'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/http\_client/pool.rb:257:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:347:in`with\_connection'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/http\_client/pool.rb:256:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:264:in`put'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/http\_client.rb:337:in `template_put'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:82:in`template\_install'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/template\_manager.rb:29:in `install'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:9:in`install\_template'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:58:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:25:in`register'", "/usr/share/logstash/logstash-core/lib/logstash/output\_delegator\_strategies/shared.rb:9:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:43:in`register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:290:in `register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:301:in`register\_plugins'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:301:in`register\_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:310:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:235:in`run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:398:in `start\_pipeline'"]} ---
**Author:** ![pope843](https://avatars.discourse-cdn.com/v4/letter/p/f4b2a3/32.png) [@pope843](https://discuss.elastic.co/u/pope843)\ **Post date:** [December 3, 2017, 1:20am UTC](https://discuss.elastic.co/t/dupilcate-message-on-elasticsearch/107182/6 "2017-12-03T01:20:29Z")
Hello Going back to Duplicate messages I still encounter it, i do have below config filter { fingerprint { source =\> "message" target =\> "%{[@metadata][fingerprint]}" method =\> "MURMUR3" } } output { if [host] == "localhost" { if [type] == "router" { elasticsearch { hosts =\> ["hostko:443"] document\_id =\> "%{[@metadata][fingerprint]}" index =\> "router-applog-%{+YYYY.MM.dd}" template =\> "/home/ubuntu/ELK/my-applogtemplate-updated.json" template\_name =\> "applog-_" template\_overwrite =\> true document\_type =\> "%{[@metadata][type]}" } } else if [type] == "syslog" { elasticsearch { hosts =\> ["hostko:443"] sniffing =\> false index =\> "maprouter-syslog-%{+YYYY.MM.dd}" template =\> "/home/ubuntu/ELK/my-syslogtemplate-updated.json" template\_name =\> "syslog-_" template\_overwrite =\> true document\_type =\> "%{[@metadata][type]}" document\_id =\> "%{[@metadata][fingerprint]}" } } else { file { path =\> "/var/log/logstash2/unknown\_messages.log" } } } } on Kibana, i can see that there are duplicates, the only difference is the "\_id". Please help ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 31, 2017, 1:20am UTC](https://discuss.elastic.co/t/dupilcate-message-on-elasticsearch/107182/7 "2017-12-31T01:20:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.