Logstash - not update a document even if document_id is specified

[reed (alucard)]

Hi all,

I'm new with elastic technology and I wold ask this:

I have my pipeline with

output
{
elasticsearch
{
hosts => "localhost:9200"
index => "crsi-index-template-%{+yyyy.MM.dd}"
document_id => "%{[transaction_id]}"
action => "update"
doc_as_upsert => true
}

stdout {}
}

yestarday I populated elastic data with three documents, so in the index "crsi-index-template-2021.09.07", today I've tried to update one of these documents with an existing "transaction_id" and logstash has created a new document insted to update it.

I can't understand why, someone can hel me on this?

regards.

[leandrojmp (Leandro Pereira)]

If I'm not wrong, the _id is not unique across indices and since you are using a time based index you seem to have 2 indices.

The first one is crsi-index-template-2021.09.07 and the second one is crsi-index-template-2021.09.08, so you didn't updated a document, you create another document with the same _id in another index.

[reed (alucard)]

Hi, thanks for the fast responde.

so if I want to update the same document I need to have always the same index?

There is no way to update the document in my configuration?

thanks.

[leandrojmp (Leandro Pereira)]

You would need to use a filter in your pipeline to query your elasticsearch using this id and get the correct index name, but depending on the number the events this could be expensive to your cluster.

What does your document looks like? I see that you have a Time field with a date, normally this is the date that should be used when create the index, so your document would be stored in the index crsi-index-template-2020.12.10 and it would update as the date is the same, but you seems to be using the @timestamp generated by logstash during ingestion.

If you could use the document date instead, this would solve your update problemas as the date would be the same and would direct to the same index.