ILM and determining the destination index without a lookup

Elastic Stack Elasticsearch
1

Hello,

I'm using ILM to automatically rollover indices monthly.

I have to bulk insert (or rather, upsert) a bunch of documents with pre-assigned ids, and I want to ensure that there won't be duplicates in different indices under the same alias (i.e. I don't want the document with the same id to be present in both the July index and the August index).

For that I wanted to build the index based on the timestamp of the document.

E.g. say I have indices like:

  • myindex-2024.08
  • myindex-2024.07
  • myindex-2024.06

and so on.

Now I get a document I want to upsert, dated somewhere in July. The document might not be there or it might have updated data.

Prior to ILM we had some custom code to rollover indices manually, so we'd just build the target index name in code based on the document date, in this case myindex-2024.07.

However the problem with ILM is that it apparently forces you to have a numeric index at the end, otherwise I get an error like:

index name [<myindex-{now/M{yyyy.MM}}>] does not match pattern '^.*-\\d+

so I have to do something like:

<myindex-{now/M{yyyy.MM}}-1>

Which means I end up with indices like:

  • mytest-2024.07-1
  • mytest-2024.08-000002

Which means I would have to know/keep track of the numerical index and I can't rely on the document date alone.

Does this mean I need to run a search to determine the destination index of the documents, with the corresponding impact in performance?

Thanks in advance.

2

This is not a limitation of ILM but rather with rollover. You can use ILM with the type of indices you created earlier, which if you need to perform upserts probably is better. I would therefore recommend you switch back to the old way of indexing with date in the index name and stop using rollover.

You can create an ILM policy without rollover to handle the lifecycle and eletion of these indices.

Yes, if you are uisng rollover you likely will need to perform a search before you update or update using update-by-query.

1 Like
3

Thank you, I'll do that.