How to reindex when source indices are continuously receiving data?

Elastic Search
1

Title:
How to reindex when source indices are continuously receiving data?

Description:
I’m trying to perform a reindex operation in Elasticsearch where the source indices are continuously receiving new data.

Scenario:
I have two source indices:

  • log-platform-postgresql-*

  • log-platform-keepalived-*

The destination index is:

  • app-platform-*

Both source indices are actively ingesting data in real time.

Issue:
When I trigger the reindex operation, it copies the existing data at that moment. However, any new data that continues to arrive in the source indices during or after the reindex process is not reflected in the destination index (app-platform-*).

Question:
What is the recommended approach to handle reindexing in this scenario where the source indices are continuously being updated?
How can I ensure that the destination index stays in sync with incoming data during or after the reindex process?

2

Why do you want to reindex the data into a new index pattern? Knowing the rationale behind why you want to duplicate the data this way this would be useful and help identify options.

3

My goal is to consolidate data from multiple source indices into a single destination index so that I can build a unified and visually rich dashboard in Kibana.

4

Are you making any changes to the data or mappings when you reindex? If not, have you considered just adding an alias on top of the indices you want to consolidate and use this for visualisations in Kibana? That would remove the need to duplicate data and would always be up to date.

2 Likes
5

Or a Data View which is exactly what they are used for... So that you can use a data view that references many indices in dashboards and visualizations and alerts and so on

1 Like
6

No, we dont change anything yet. but in future we will try add labels in the data.

7

In that case I would recommend adding the labels before ingesting it into the original index and then use an alias or a Data View as described above. I see no point in reindexing the data.