Hello everyone!
Suddenly, the elastic began to double logs. Each log falls into its own index and the duplicate falls into the unkown_messages index. It seems that this started after adding new indexes, but rolling back the config did not help.
LS configs are stored in /etc/logstash/conf. d/, there are 3 files:
00-input. conf
10-filter.conf
20-output.conf
20-output is like this:
output {
Configuring Windows Domain Controllers log output
if "windc" in [tags] {
elasticsearch {
hosts => ["10.199.5.104:9200","10.199.5.105:9200","10.199.5.106:9200"]
index => "windc-%{+YYYY.MM.dd}"
}
}Configuring ksmg log output
if "ksmg" in [tags] {
elasticsearch {
hosts => ["10.199.5.104:9200","10.199.5.105:9200","10.199.5.106:9200"]
index => "ksmg-%{+YYYY.MM.dd}"
}
}Configuring the output of Exchange mail logs
if "exchange-mtlog" in [tags] {
elasticsearch {
hosts => ["10.199.5.104:9200","10.199.5.105:9200","10.199.5.106:9200"]
index => "exchange-mtlog-%{+YYYY.MM.dd}"}
}Configuring the output of other logs
else {
elasticsearch {
hosts => ["10.199.5.104:9200","10.199.5.105:9200","10.199.5.106:9200"]
index => "unknown_messages"
}
}}
After running logstash, the service log contains these messages::
[WARN ][logstash.outputs.elasticsearch][main][e9b556ef36eb5a3aa1e07673fcd36a804aeecfb9a6de701dcf08a41e172da77a] Could not index event to Elasticsearch. {:status=>400,:action=>["index", {:_id=>nil, :_index=>"unknown_messages", :routing=>nil, :_type=>"_doc"},#LogStash::Event:0x682b1390], :response=>{"index"=>{"_index"=>"unknown_messages","_type"=>"_doc", "_id"=>"GcUcXnQBuxL4OHJpyi4Y", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [source] of type [text] in document with id 'GcUcXnQBuxL4OHJpyi4Y'". Preview of field's value: '{domain=DPC-RDS}'", "caused_by"=>{"type"=> "illegal_state_exception", "reason"=> " Can't get text on a START_OBJECT
Preview of field's value: '{domain=DPC-RDS}'" - this part changes depending on the incoming log, the rest is identical. Ah and ID respectively, too, different. And logs with different IDS end up in indexes. The tags for which logs should fall into the corresponding indexes are found in both the correct indexes and the unknown_messages index.
Has anyone ever encountered this? Which way to dig?