Avoid hiting unecessary shards with wildcard dated index

Elastic Stack Elasticsearch
1

I have multiple indices with a date in the name that stores data for specific days for each months, so I have day-2024.01.01, day-2024.02.01 and so on. The indices are managed by ILM and I always have 2 years of future data, I have a cron that creates the new month in the future each months.

The issue is, when doing search or updateByQuery on wildcard index (day-*), I might have a no_shard_available_exception when the new indices are being created. This is always a very temporary issue as the shard are being allocated fast enough, but I would like to avoid this issue if possible. This last index is never used during the month it is created, it will start to index data the next month so I create it very early.

The thing is then that Elasticsearch doesn't really need the newly created index when doing thoses queries but it doesn't know that at the step when it checks if the shards are available. All my queries have a range filter on the date of the days that will always exclude potential data in the last index, so I would like to know if there is a way to limit the wildcard to indices that match the date range filter? Like a setting to tell elasticsearch "this index stores only from date X to X so ignore this index when the query match another date"?

I think of two things that could work in my cases but would require some extra work on my backend that I would like to avoid if a better solution exists:

  • Avoid wilcard and calculate very indices I need from my backend and run the queries specifically on them
  • Exclude the last index only when doing my queries when the new index is new (like day-*,-day-2026.04.01)
2

I would suggest adding an alias that you use to search your data. This will map to indices similar to when using an index pattern with wildcard but allow you to control when the new index is added to the alias during the creation period. It adds a step to the creation proces, but should resolve the transient issue.

3

Thank you very much!
I already have an alias but it is set at the same time that the index is created (defined in the template). I'll make thoses changes and have my creation script monitor the allocation and add the alias when its done.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.