Logs duplication

Hello everyone!

Suddenly, the elastic began to double logs. Each log falls into its own index and the duplicate falls into the unkown_messages index. It seems that this started after adding new indexes, but rolling back the config did not help.

LS configs are stored in /etc/logstash/conf. d/, there are 3 files:
00-input. conf
10-filter.conf
20-output.conf

20-output is like this:

output {

Configuring Windows Domain Controllers log output

if "windc" in [tags] {
elasticsearch {
hosts => ["10.199.5.104:9200","10.199.5.105:9200","10.199.5.106:9200"]
index => "windc-%{+YYYY.MM.dd}"
}
}

Configuring ksmg log output

if "ksmg" in [tags] {
elasticsearch {
hosts => ["10.199.5.104:9200","10.199.5.105:9200","10.199.5.106:9200"]
index => "ksmg-%{+YYYY.MM.dd}"
}
}

Configuring the output of Exchange mail logs

if "exchange-mtlog" in [tags] {
elasticsearch {
hosts => ["10.199.5.104:9200","10.199.5.105:9200","10.199.5.106:9200"]
index => "exchange-mtlog-%{+YYYY.MM.dd}"

}
}

Configuring the output of other logs

else {
elasticsearch {
hosts => ["10.199.5.104:9200","10.199.5.105:9200","10.199.5.106:9200"]
index => "unknown_messages"
}
}

}

After running logstash, the service log contains these messages::

[WARN ][logstash.outputs.elasticsearch][main][e9b556ef36eb5a3aa1e07673fcd36a804aeecfb9a6de701dcf08a41e172da77a] Could not index event to Elasticsearch. {:status=>400,:action=>["index", {:_id=>nil, :_index=>"unknown_messages", :routing=>nil, :_type=>"_doc"},#LogStash::Event:0x682b1390], :response=>{"index"=>{"_index"=>"unknown_messages","_type"=>"_doc", "_id"=>"GcUcXnQBuxL4OHJpyi4Y", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [source] of type [text] in document with id 'GcUcXnQBuxL4OHJpyi4Y'". Preview of field's value: '{domain=DPC-RDS}'", "caused_by"=>{"type"=> "illegal_state_exception", "reason"=> " Can't get text on a START_OBJECT

Preview of field's value: '{domain=DPC-RDS}'" - this part changes depending on the incoming log, the rest is identical. Ah and ID respectively, too, different. And logs with different IDS end up in indexes. The tags for which logs should fall into the corresponding indexes are found in both the correct indexes and the unknown_messages index.

Has anyone ever encountered this? Which way to dig?

What you have is

if "windc" in [tags] {
}
if "ksmg" in [tags] {
}
if "exchange-mtlog" in [tags] {
} else {
}

and it sounds like what you want is

if "windc" in [tags] {
} else if "ksmg" in [tags] {
} else if "exchange-mtlog" in [tags] {
} else {
}
2 Likes

Yes, you right. I changed it to be same as in filter. conf for no reason...Thank you very much!