Update document on multiple indices

Elastic Stack Logstash
1

Hi,

I'm tracking user sessions as docs in ES.
Each session is a single doc that gets updated when ever the session state changes.

I'm using daily based indices, and the ID of the doc is the session ID (UUID).
If a session starts before the end of a day, but continues on to the next day, the update will look for it only in the next day's index and thus won't find it and will create a new doc, right?

How can I make logstash look up the doc in multiple indices so I will really only have a single doc per session? Or is there a better way to handle this?

2

Hi shaharmor,

i have an Idea, but it's a little bit complicated. I think the decission in which Index the event is written depends on the timestamp of the doc or other specific fields.

To solve your Problem you can do this:

  1. You lookup your ES if there is already an entry with the same UUID. You can do this with the Elasticsearch-Filter-Plugin (here the documentation: https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html).

  2. Configure the filter plugin, so that you "join" specific fields (that are responsible for writing in the time based index) from the existing entry in ES. If there is no existing entry with the same UUID, the filter plugin doesn' join anything (because ther is nothing to join) and the doc won't be modified.

  3. If the filter plugin joined the fields of the existing entry, delete the unnecessary and keep the necessary fields for the correct updating to the "old" index.

I hope i could help you a little bit =)

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.