Duplicate logs

Elastic Stack Elasticsearch
1

Hello,

I have ELK version 6.2.2. Configured and working..
I have installed ELK in two nodes.
I observed that Kibana started showing duplicate logs... as in the picture..

Duplicate1
Duplicate1.PNG1453×139 9.83 KB

Duplicate2
Duplicate2.PNG882×905 48 KB

Below is my Logstash configuration

input {
            file{
        		path => ["<-- file path --> "]
        		start_position => "beginning"
        		sincedb_path => "/appl/log/sincedbloc/sincedbfile.txt"
        		codec => json
        	}
    }

filter {
	date{
		locale => "en"
		match => ["loggingTime","EEE d MMM yyyy HH-mm-ss SSS z"]
		target => "@timestamp"
	}
	
	fingerprint {
		source => ["@timestamp","logMessage"]
		target => "fingerprint"
		key => "78787878"
		method => "SHA1"
		concatenate_sources => true
	}	
}

output {
	elasticsearch { 
  		hosts => ["HOST1-DNS-NAME:9200","HOST2-DNS-NAME:9200"]
  		document_id => "%{fingerprint}"
  		index => "logstash-%{+YYYY.MM.dd}"
	}
}

Could you help whats wrong.. ?

2

I have given
document_id => "%{fingerprint}" in the output plugin..

But still elasticsearch produces an new _id value and duplicates logs.. as I have marked in the attached picture.
Which I don't understand ..Why ?
Am I missing anything?

3

Do you have any other configuration files that could get picked up and does not set the document id? If you search for a sample fingerprint in the fingerprint field, do you get more than one hit?

4

Thanks for the response Christian.
I don't have any other configuration to pickup the same log message.
I'm facing duplicate logs only in my QA ELK servers.. I have Prod ELK server with exact same configuration but I'm not facing any duplicate logs problem.. The only difference is, in QA I have ver 6.2.2 in Prod I have ver 2.4.0. I'm planning to upgrade Prod from 2.4.0 to 6.2.2.. But stopped by this duplicate message issue.

again I have attached screen shots for more information to you from few logs

Duplicate3
Duplicate3.PNG1515×515 45.3 KB
.

Expanded view of a duplicate

Duplicate4
Duplicate4.PNG1146×956 60.1 KB

5

Given that you have duplicates, it still sounds like you have multiple configuration files. Did you install Logstash as a service? If so, what is the full content of its config directory?

6

Yes I have installed Logstash as Service.. here is the configuration file that I have for logstash

input {
    	file{
    		path => ["/dhmi1was1_logs/GALCServerLogs/server-jsonoutput.log","/dhmi1was2_logs/GALCServerLogs/server-jsonoutput.log","/qhmi1was1_logs/GALCServerLogs/server-jsonoutput.log","/qhmi1was2_logs/GALCServerLogs/server-jsonoutput.log","/qhmi1was3_logs/GALCServerLogs/server-jsonoutput.log","/qhmi1was4_logs/GALCServerLogs/server-jsonoutput.log"]
    		start_position => "beginning"
    		sincedb_path => "/appl/log/sincedbloc/sincedbfile.txt"
    		codec => json
    	}
}

filter {
	date{
		locale => "en"
		match => ["loggingTime","EEE d MMM yyyy HH-mm-ss SSS z"]
		target => "@timestamp"
	}
	
	fingerprint {
	        id => "ApplicationLogs"
	        source => ["@timestamp","logMessage"]
		target => "[fingerprint]"
		key => "78787878"
		method => "SHA1"
		concatenate_sources => true
	}	
}

output {
	elasticsearch { 
  		hosts => ["HOST1:9200"]
  		document_id => "%{[fingerprint]}"
  		index => "logstash-%{+YYYY.MM.dd}"
	}
	file{
		path => "/log/logstash/Logstash_Output.log"
	}
}
7

Are there ANY other files in the directory where this config file resides?

8

There are 2 more files along with logstash.conf in its conf.d directory..

  1. logstash.conf.orig .. Has the exact same as logstash.conf ( .orig are getting created by logstash itself?)

  2. logstash-simple.conf.. has following plugin content in it..

    input { stdin { } }
    output {
    elasticsearch { hosts => ["localhost:9200"] }
    stdout { codec => rubydebug }
    }

9

Those files will all get concatenated so could explain your duplicates.

Does logstash.conf.orig also have document id set to the fingerprint field?

10

Yes. Logstash.conf.orig is exact copy of Logstash.conf

11

Do you have a local Elasticsearch node on the host where Logstash runs?

I would recommend removing these two other files from that directory and see if that causes duplicates to stop being produced.

12

Yes.. I have ..
I have a server.. where one instance of Logstash , Elasticsearch and Kibana is running..

Okay.. I will remove it and check again..

13

Then it is the logstash-simple.conf file that is causing the duplicates as it does not set document id.

14

Yes.. that fixed the problem.. I have only logstash.conf under conf.d directory..
Now no more duplicates.

Thank you so much.. problem solved.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.