Backfill with previous indexed data


When the document sorted by timestamp, is there any solution that backfill location data into next row with previous row ???

You mean you want to merge those two rows you have ticked?

Those are different logs.
Either merge or copy are fine!

It's not clear what you mean sorry.

I think it's a #elastic-stack:kibana question, right?
You don't mean to change the indexed data, correct?

I don't think it's doable.

The only way to do this IMO is to solve that problem at index time.
If the data is empty, run a search, get the hit which is meant to be just before, extract the location, set the location value of the current event with this extracted value, send the document to Elasticsearch.

real log created by each system so the each row could be separated.
Logstash was used for indexing. Is there any way on creating the index, fetch the previous data?....
I think there should be a function on kibana or dev tool not on the logstash if there are no features for these backfill.

You can open a feature request for it. Not sure how this will be handled though. :slight_smile:

You can do a lookup in Elasticsearch, using the Elasticsearch filter plugin. And activate this plugin with some Logstash conditionals like an empty location field...
Not an easy task IMO but doable hopefully...

If you need advices for building this, I'd suggest to ask in Logstash channel...

I resolved this issue with Aggregate filter (Aggregate filter plugin | Logstash Reference [8.7] | Elastic)

My case was different with official document example cases,
so I had to use task_id is static one (such as %{host}%{file} )

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.