Timeseries data with ILM and out of sequence write/delete of older data

Elastic Stack Elasticsearch
1

Hi, this question is a follow on from a Previous post.

I am looking to merge many indices into 1 and using ILM to manage index size etc. However what happens if I have to update or delete a document in an index that is no longer the hot index? Can I keep all indices as hot, but still use ILM to keep their sizes reasonable?

I have time based data that gets uploaded in blocks. Sometime blocks are missed/lost or fail to upload and get recovered days/weeks later. In this case I need to go insert the data. Presumably data insertion is not a problem, even if out of sequence as it will just write to the current index? Or does it try keep the index sorted based on the indexed fields?

But what happened if I need to up/delete data that has already been moved to another index. The only way I think I can do that is to keep the indices hot?

What am I missing, is there a better way of managing this? The reason I am wanting to merge the indices into 1 is because I have far too many (1800+). So as discussed in the "Previous post" I should move these to a single index and use ILM...

Thanks, your thoughts are appreciated.

2

You need to find the index that the document lives in and update it there, not via the alias.

The only limitation I can think of would be that the data may be deleted after your logical retention period, as it will be older than the actual index age.

3

Hi Mark, thank you for the reply. If I only set a hot phase in the ILM with roll over. Would that allow me to update documents via the alias? Or is the problem that an older index is no longer the write index and therefore I can't update/delete through the alias?

4

An alias can only be used to update or delete documents if it is only backed by a single index. You can not update against an alias that point to multiple indices.

5

OK thanks....

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.