# Logstash Duplicate message **URL:** \ **Category:** Logstash\ **Created:** [July 22, 2015, 3:43pm UTC](https://discuss.elastic.co/t/logstash-duplicate-message/26096 "2015-07-22T15:43:27Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Junior\_Burleon](https://avatars.discourse-cdn.com/v4/letter/j/e47774/32.png) [@Junior\_Burleon](https://discuss.elastic.co/u/Junior_Burleon)\ **Post date:** [July 22, 2015, 3:43pm UTC](https://discuss.elastic.co/t/logstash-duplicate-message/26096/1 "2015-07-22T15:43:27Z")
Hi, I really need your help. I use LogStash-Forwarder on a server to send my log on another server that use LogStash. And I Visualize all of my logs with Kibana. (Of course a use ElasticSearch for the analyze but I think is not the problem here) * * * 1. My first problem is that my log between my log source file and Kibana are duplicate. I tried many solution that I find on the web but nothing resolve my issue. 2. My second issue is why my TimeStamp in Kibana is with +2hours than the TimeStamp in my log file ? * * * ## /etc/logstash-forwarder.conf ``` { "network": { "servers": ["192.168.250.65:5000"], "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt", "timeout": 15 }, "files": [ { "paths": [ "/data/jboss/jboss-fuse-6.0.0.redhat-024/data/log/monitoring.lo*" ], "fields": { "type": "transporter" } } ] } ``` ## /etc/logstash/conf.d/01-lumberjack-input.conf ``` input { lumberjack { port => 5000 type => "logs" ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" } } ``` ## /etc/logstash/conf.d/11-transporter.conf ``` filter { if [type] == "transporter" { grok { match => [ "message", "%{MESSAGE}", "message", "%{MESSAGE_PLAN}", "message", "%{MESSAGE_OLD}" ] } } } ``` ## /etc/logstash/conf.d/30-lumberjack-output.conf ``` output { if [type] == "transporter" { elasticsearch { host => "localhost" index => "transporter-%{+YYYY.MM.dd}" template => "/etc/logstash/mapping/transporter.json" template_name => "transporter" template_overwrite => true } } else { elasticsearch { host => "localhost" } stdout { codec => rubydebug } } } ``` ## My pattern are describe here ``` MESSAGE_OLD %{GREEDYDATA:log_date} \| Message:\|%{GREEDYDATA:message_type}\|:\[%{GREEDYDATA:component}]:%{GREEDYDATA:scope};%{GREEDYDATA:issue_date};%{GREEDYDATA:title};%{GREEDYDATA:edition};%{GREEDYDATA:file_name}\|%{GREEDYDATA:message_detail} MESSAGE %{GREEDYDATA:log_date} \| Message:\|%{GREEDYDATA:message_type}\|:\[%{GREEDYDATA:component}]:%{GREEDYDATA:scope};%{GREEDYDATA:issue_date};%{GREEDYDATA:title};%{GREEDYDATA:edition};%{GREEDYDATA:file_name};%{GREEDYDATA:file_type};page:%{GREEDYDATA:page_number}\|%{GREEDYDATA:message_detail} MESSAGE_PLAN %{GREEDYDATA:log_date} \| %{GREEDYDATA:message_type}\:%{GREEDYDATA:scope};%{GREEDYDATA:issue_date};%{GREEDYDATA:title};%{GREEDYDATA:edition};\[Pages:%{NUMBER:pages},Articles:%{NUMBER:articles},Images:%{NUMBER:images} ``` ## Example of my log monitoring.log ``` 22 Jul 2015 17:30:02,684 | Message:|Info|:[ComparechksProducer]:afp;;;;20150722142803_FGJ33.xml;;page:|The file not exists the entryChecksum collection. 22 Jul 2015 17:30:02,685 | Message:|Info|:[comparechks]:afp;;;;fra/20150722142803_FGJ33.xml;;page:|false 22 Jul 2015 17:30:02,711 | Message:|Info|:[ComparechksProducer]:afp;;;;20150722141409_FGI57.xml;;page:|The file not exists the entryChecksum collection. 22 Jul 2015 17:30:02,712 | Message:|Info|:[comparechks]:afp;;;;fra/20150722141409_FGI57.xml;;page:|false 22 Jul 2015 17:30:02,735 | Message:|Info|:[ComparechksProducer]:afp;;;;20150722141908_FGI88.xml;;page:|The file not exists the entryChecksum collection. 22 Jul 2015 17:30:02,736 | Message:|Info|:[comparechks]:afp;;;;fra/20150722141908_FGI88.xml;;page:|false ``` ## The same log in Kibana ![](https://us1.discourse-cdn.com/elastic/original/2X/d/d41f9d288966216fcbfe04161f5c38cd275975bb.png) ## My template for ElasticSearch /etc/logstash/mapping/transporter.json ``` { "template": "transporter-*", "settings": { "index.refresh_interval": "10s", "index": { "number_of_shards": 3, "number_of_replicas": 1 } }, "mappings": { "_default_": { "dynamic_templates": [ { "message_field": { "mapping": { "index": "analyzed", "omit_norms": true, "type": "string" }, "match": "message", "match_mapping_type": "string" } }, { "string_fields": { "mapping": { "index": "analyzed", "omit_norms": true, "type": "string", "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } } }, "match": "*", "match_mapping_type": "string" } } ], "_all": { "enabled": true, "omit_norms": true }, "properties": { "@version": { "type": "string", "index": "not_analyzed" }, "geoip": { "dynamic": "true", "properties": { "location": { "type": "geo_point" } } } } }, "transporter": { "dynamic_templates": [ { "message_field": { "mapping": { "index": "analyzed", "omit_norms": true, "type": "string" }, "match": "message", "match_mapping_type": "string" } }, { "string_fields": { "mapping": { "index": "analyzed", "omit_norms": true, "type": "string", "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } } }, "match": "*", "match_mapping_type": "string" } } ], "_all": { "enabled": true, "omit_norms": true }, "properties": { "@timestamp": { "type": "date", "format": "dateOptionalTime" }, "@version": { "type": "string", "index": "not_analyzed" }, "articles": { "type": "integer" }, "component": { "type": "string", "index": "not_analyzed" }, "edition": { "type": "string", "index": "not_analyzed" }, "file": { "type": "string", "index": "not_analyzed" }, "file_name": { "type": "string", "index": "not_analyzed" }, "file_type": { "type": "string", "index": "not_analyzed" }, "log_date": { "type": "date", "format": "dd MMM yyyy HH:mm:ss,SSS", "index": "not_analyzed" }, "host": { "type": "string", "index": "not_analyzed" }, "images": { "type": "integer" }, "issue_date": { "type": "string", "index": "not_analyzed" }, "message": { "type": "string", "index": "not_analyzed" }, "message_detail": { "type": "string", "index": "not_analyzed" }, "message_type": { "type": "string", "index": "not_analyzed" }, "page_number": { "type": "string", "index": "not_analyzed" }, "pages": { "type": "integer" }, "scope": { "type": "string", "index": "not_analyzed" }, "title": { "type": "string", "index": "not_analyzed" }, "type": { "type": "string", "index": "not_analyzed" } } } } } ``` Thanks for your help and if you need others informations, don't hesitate to ask me !! ---
**Author:** ![Junior\_Burleon](https://avatars.discourse-cdn.com/v4/letter/j/e47774/32.png) [@Junior\_Burleon](https://discuss.elastic.co/u/Junior_Burleon)\ **Post date:** [July 23, 2015, 11:30am UTC](https://discuss.elastic.co/t/logstash-duplicate-message/26096/2 "2015-07-23T11:30:43Z")
Nobody have an idea to help me ? ---
**Author:** ![SNAFUdowser](https://avatars.discourse-cdn.com/v4/letter/s/82dd89/32.png) [@SNAFUdowser](https://discuss.elastic.co/u/SNAFUdowser)\ **Post date:** [July 23, 2015, 1:59pm UTC](https://discuss.elastic.co/t/logstash-duplicate-message/26096/3 "2015-07-23T13:59:33Z")
For duplicate messages, try using this: filter { ``` uuid { target => "@uuid" overwrite => true } fingerprint { source => ["message"] target => "fingerprint" key => "78787878" method => "SHA1" concatenate_sources => true } } output { elasticsearch { host => localhost document_id => "%{fingerprint}" } stdout { codec => rubydebug } } ``` This will create a hash of your message and then if there is an exact duplicate then it will overwrite it. (In case you don't want two logs with the same contents at different times, you can use mutate{} [https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html) to remove the timestamp) As for the timestamp, try using date {} [http://stackoverflow.com/questions/26035136/logstash-custom-date-log-format-match](http://stackoverflow.com/questions/26035136/logstash-custom-date-log-format-match) ---
**Author:** ![Junior\_Burleon](https://avatars.discourse-cdn.com/v4/letter/j/e47774/32.png) [@Junior\_Burleon](https://discuss.elastic.co/u/Junior_Burleon)\ **Post date:** [July 27, 2015, 2:06pm UTC](https://discuss.elastic.co/t/logstash-duplicate-message/26096/4 "2015-07-27T14:06:59Z")
I used document\_id =\> "%{fingerprint}" And after I suppress this line, and now it's ok, my logs are not duplicate. I don't understand why. But if someone have this problem try with add it. (me I suppress it because it clean too much my log, and after suppress this line, It was just ok. And for my problems with the time, it was because my log doesn't have the time zone. I'm in GMT+2 so when ElasticSearch parse the date without the TimeZone, it doesn't care about it. I explain what is my solution, maybe it can help someone. Thanks for your help ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:33am UTC](https://discuss.elastic.co/t/logstash-duplicate-message/26096/5 "2017-07-06T05:33:46Z")