Hello!
I use data streams for store data. And I have data stream alias. How I can automatically delete back indexes older 7 days from alias?
Please have a look at using ILM policies to automate this!
I need save index, but It must be removed from alias
Can you explain a bit more about what you're trying to achieve?
Can this be solved by using a ranger filter instead, to limit your searches to the data from the last 7 days (regardless of which index they live in behind the alias)?
I use alias for SIEM rules. And now indices have dynamic mappings. Sometimes there are Siem rules errors occur due to mapping conflicts. Now I solve this problem by changing the data stream name and adding it to alias. But I need alias consists only last backing data stream index.
P.S. I know that strict mapping is the best decision. Gradually I use it
Right, so you have a SIEM rule looking at data from the last 7 days.
Due to the mapping change you now have some indices with one mapping, and the newer ones have a different mapping causing the conflict when the rule runes since the rule looks at both type of indices and the mappings conflict.
So rather than waiting for 7 days for the old indices to move out of the look back window of the rule, you want to remove those old indices from the alias? The risk of doing that is that your rule won't be looking at that data anymore (defeating the point of the look back window you set).
If that's okay for you, isn't it possible to simply remove the alias for those old indices?
Another option might be to re-index those old indices so they get the new mappings.
Unfortunately, I can't remove separate back index from alias. I only can remove full data stream.
And I can't reindex old data because it will cause cluster overload.
Therefore, I want to automatically delete indexes older than a 7 days
For this particular case, would it be enough to only delete the indices that are within the last 7 days which have the old mapping?
That should make the SIEM rule work going forward.
I need save index, but remove from alias. It is good way for SIEM rules
I got a tip from the Elasticsearch devs that there is an API to modify a data stream that would allow you to exclude the backing indices from the data stream without deleting them.
1 Like
Нос I can automatically exclude back index? May be I can use ILM for such purpose?
As far as I know you can not do this automatically, and likely not even with ILM as there is no appropriate action. I suspect you will need to manage this yourself externally to Elasticsearch.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.