Hello,
FileBeat 6.6.2
LogStash 6.6.2
ElasticSearch 5.6.10
I am trying to combine data from 2 different log files into the same document in ElasticSearch.
I read each log file with a separate instance of FileBeat, both sending output to the same LogStash instance.
Each FileBeat config file sets a different value for custom field
fields: {log_type: blah}
In that LogStash instance, I process the input differently based on the value of [fields][log_type] and, in both cases, its sends the output to ElasticSearch, with the same index for related pieces of data.
Each processing section updates different fields in ElasticSearch, except the "index" and the "document_id".
However, what I see in Kibana monitor is that different documents are being created. They have the same value of field "_id", but they haven't been merged.
I have this in my elasticsearch output plugin
action => "update"
doc_as_upsert => true
What am I missing here?
Could it be because the data from the second log file is being read just a few seconds after reading the first one?
