Upsert not working for same index in Logstash output to ElasticSearch

Elastic Stack Logstash
1

Hi,

I am using same index "index => "$logstash-%{+YYYY.MM.dd}"" for multiple configuration reading multiple logs so as to have one index for certain type of logs.

It only writes index in ElasticSearch for 1st configuration file which reads the logs and not for the other ones.

For example: I had an implementation with single configuration and index named "$logstash-%{+YYYY.MM.dd}"". Now I added another configuration to read other files and used same index "$logstash-%{+YYYY.MM.dd}"". In new configuration, filter generated more fields which didn't get updated in existing index of Elastic Search though it showed on stdout.

Output settings:

output {
elasticsearch {
hosts => ["ES01:9200"]
index => "$logstash-%{+YYYY.MM.dd}"
}
}

I saw two options to use doc_as_upsert => "true" and document_id field. I am not sure if document_id should be same in both configurations. please help.

How to upsert new fields so as same index works for multiple configuration (fields generated from filter)?

2

Hi, Root cause of issue has been identified.
The Upsert worked fine with multiple such setups except the one config which had input filter having ".log" for reading all files from one log folder. The same was adjusted as it was found other configs with unique input file name worked perfectly fine. To address this, I gave all names in input file instead of generic "c:/somedrive/.log" and the same worked fine.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.