# Backfill with previous indexed data

**URL:** <https://discuss.elastic.co/t/backfill-with-previous-indexed-data/329278>\
**Category:** Logstash\
**Created:** [April 4, 2023, 7:01am UTC](https://discuss.elastic.co/t/backfill-with-previous-indexed-data/329278 "2023-04-04T07:01:25Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![suminlim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/suminlim/32/97487_2.png) [@suminlim](https://discuss.elastic.co/u/suminlim)\
**Post date:** [April 4, 2023, 7:01am UTC](https://discuss.elastic.co/t/backfill-with-previous-indexed-data/329278/1 "2023-04-04T07:01:25Z")

</div>

![problem01](https://us1.discourse-cdn.com/elastic/original/3X/0/1/01c8e5079fc6f6e1eafee04d8f858029a71f1e46.png)  
When the document sorted by timestamp, is there any solution that backfill location data into next row with previous row ???

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [April 4, 2023, 7:11am UTC](https://discuss.elastic.co/t/backfill-with-previous-indexed-data/329278/2 "2023-04-04T07:11:12Z")

</div>

You mean you want to merge those two rows you have ticked?

---

<div class="post-metadata">

**Author:** ![suminlim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/suminlim/32/97487_2.png) [@suminlim](https://discuss.elastic.co/u/suminlim)\
**Post date:** [April 4, 2023, 7:12am UTC](https://discuss.elastic.co/t/backfill-with-previous-indexed-data/329278/3 "2023-04-04T07:12:36Z")

</div>

Those are different logs.  
Either merge or copy are fine!

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [April 4, 2023, 7:17am UTC](https://discuss.elastic.co/t/backfill-with-previous-indexed-data/329278/4 "2023-04-04T07:17:31Z")

</div>

It's not clear what you mean sorry.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 4, 2023, 7:26am UTC](https://discuss.elastic.co/t/backfill-with-previous-indexed-data/329278/5 "2023-04-04T07:26:30Z")

</div>

I think it's a #elastic-stack:kibana question, right?  
You don't mean to change the indexed data, correct?

I don't think it's doable.

The only way to do this IMO is to solve that problem at index time.  
If the data is empty, run a search, get the hit which is meant to be just before, extract the location, set the location value of the current event with this extracted value, send the document to Elasticsearch.

---

<div class="post-metadata">

**Author:** ![suminlim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/suminlim/32/97487_2.png) [@suminlim](https://discuss.elastic.co/u/suminlim)\
**Post date:** [April 4, 2023, 7:41am UTC](https://discuss.elastic.co/t/backfill-with-previous-indexed-data/329278/6 "2023-04-04T07:41:05Z")

</div>

real log created by each system so the each row could be separated.  
Logstash was used for indexing. Is there any way on creating the index, fetch the previous data?....  
I think there should be a function on kibana or dev tool not on the logstash if there are no features for these backfill.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 4, 2023, 8:40am UTC](https://discuss.elastic.co/t/backfill-with-previous-indexed-data/329278/7 "2023-04-04T08:40:48Z")

</div>

> [@suminlim](#):
>
> I think there should be a function on kibana

You can open a feature request for it. Not sure how this will be handled though. 🙂

> [@suminlim](#):
>
> Logstash was used for indexing. Is there any way on creating the index, fetch the previous data?

You can do a lookup in Elasticsearch, using the Elasticsearch filter plugin. And activate this plugin with some Logstash conditionals like an empty `location` field...  
Not an easy task IMO but doable hopefully...

If you need advices for building this, I'd suggest to ask in #Logstash channel...

---

<div class="post-metadata">

**Author:** ![suminlim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/suminlim/32/97487_2.png) [@suminlim](https://discuss.elastic.co/u/suminlim)\
**Post date:** [April 7, 2023, 3:01am UTC](https://discuss.elastic.co/t/backfill-with-previous-indexed-data/329278/8 "2023-04-07T03:01:09Z")

</div>

I resolved this issue with Aggregate filter ([Aggregate filter plugin | Logstash Reference [8.7] | Elastic](https://www.elastic.co/guide/en/logstash/8.7/plugins-filters-aggregate.html))

My case was different with official document example cases,  
so I had to use task\_id is static one (such as %{host}%{file} )

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 5, 2023, 3:01am UTC](https://discuss.elastic.co/t/backfill-with-previous-indexed-data/329278/9 "2023-05-05T03:01:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
