Reindexing Logstash indices to use new index template

[vanessa]

Hi. We have some existing logstash logs in our elasticsearch cluster. Their pattern is logstash-imu-logs-*. We have a template for that pattern, but unfortunately it was missing the mapping for one of our fields. As the docs state, we can't retroactively apply changes we make to the logstash-imu-logs-* index template to our indices. So we want to reindex, but not clear on how to do that in that case. We've been looking at the documentation to reindex an existing index to a new index, but what about existing logs to a new index template? We thought perhaps we'd create a new template called logstash-imu-logs-reindexed-* with all the required mapping and settings, but now we don't know how to proceed from here.

[Vinayak_Sapre]

Vanessa,
Depending on type of data you need to employ different strategies. Can you provide more information on the data

1. Are indices append only? or does logstash updates existing log entries?
2. Are you using time based indices? Key characteristic is logstash creates new indices daily/weekly/monthly and older indices do not get updated.
3. If logstash updates docs, do log entries have the last updated timestamp which is updated by logstash at the time of update?

For typical append only and time based indices

1. Update template so future indices will be created with correct mapping
2. Take snapshot of old (non-updatable) indices
3. Create a new empty index logstash-imu-logs-reindexed-XYZ for existing index logstash-imu-logs-XYZ. Assuming existing template pattern matches the new index name it will have revised mapping. curl -XPUT http://localhost:9200/logstash-imu-logs-reindexed-XYZ
4. Use reindex API to reindex https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html
5. Verify all data copied. Verify snapshot in step #2 has the old index.
6. Drop the old index
7. Create alias index logstash-imu-logs-XYZ for the index logstash-imu-logs-reindexed-XYZ
8. Wait for the current index to roll over. Then follow same steps for this index.

[vanessa]

Hi @Vinayak_Sapre - apologies for the very late reply. I was working on some other things. I should mention we are using Elastic Cloud.

1. Yes, indices are append only
2. We have one index per day
3. n/a

We did do something similar to what you described, but found that not all of our indices got reindexed. We also noticed that there were fewer documents in the reindexed indices - but that's probably a separate issue where our Logstash trial and error generated duplicates. Do you know why not all indices would be reindexed? Do the steps you describe have to be done one index at a time?

[Vinayak_Sapre]

@vanessa

@vanessa

Do you know why not all indices would be reindexed? Do the steps you describe have to be done one index at a time?

Depending on how much your cluster can handle you can reindex multiple indices in parallel. See Elastic recommendation here Reindex API | Elasticsearch Guide [8.11] | Elastic

@vanessa

We also noticed that there were fewer documents in the reindexed indices - but that's probably a separate issue where our Logstash trial and error generated duplicates.

If your source has duplicates your destination should have duplicates too. Count should match. Count mismatch indicate error occurred while copying some documents or new documents got added to the source index after reindexing started. Check elasticsearch logs for reindexing errors.