Hi everyone,
I have the next scenario:
- One application log that is being loaded to ES through logstash (insert operation), the id is unique and extracted of the log and taken as _id in ES
- Other application log that is being loaded to ES through logstash (upsert operation), the id's are the same from the previous app log because of the logic of our application to keep track of the transactions, same as previous log is taken as _id in ES
the problem is that when in application1 log is near the end of the day (for example 17:59:59 becauce I'm GMT-6) it saves the documents in the day index (I have a time based index) for example logstash-2017.08.07 and normally the next step of the transaction (application2 log) is received in logstash after (18:00:00 GMT-6) so it's "duplicated" in the next day index (logstash-2017.08.08). So at the end I have the same _id in two different index
What can I do to update the first document and not generate a new one?
This is my logstash output configuration:
output {
elasticsearch{
hosts => [ "localhost:9200" ]
document_id => "%{transactionid}"
action => "update"
doc_as_upsert => "true"
}
}