How to find an index that was rolled over

soad20000 · June 5, 2024, 8:31pm

Hello, I am trying to query an old index but I can't seem to find it. The index I am looking for is " .ds-logs-system.security-default-2024.04.15-000010" which I have found a record of in the ILM history index. The record indicates that the index was rolled over and that it completed the rollover. Which from my understanding means that the index can still be queried, it just can't be written to.

This is the new index that was created but it does not contain the data I am looking for. The search should have pulled up the older index because I haven't configured any aliases for that index but it's only pulling up the new one. This is the query I used

GET _cat/indices/.ds-logs-system.security*?v

The ILM policy applying to that index has the delete phase set at 365 days.

Just to confirm the results from the API here are the search results from the GUI.

Anyone have any idea where the index went? Am I not searching for it correctly?

Thanks for reading!

soad20000 · June 11, 2024, 9:30pm

Bumping for visibility, I think sufficient time has passed. Could anyone point me in the right direction please?

leandrojmp · June 12, 2024, 3:32am

If running GET _cat/indices/ does not show the index, then it was deleted.

It was deleted manually or it is not using the policy you think it is using, but if the indice does not show up when using _cat/indices it means that the index does not exist anymore in your cluster.

Can you go into Index Management > Data Streams, look for logs-system.security-default data stream and share the pop-over information side panel that will open?

soad20000 · June 12, 2024, 3:21pm

Hi, thanks for getting back to me.

Here is the info on the data stream:

Clicking the linked lifecycle policy shows that it should be deleted after 365 days and the applied index template confirms that.

Maybe it was deleted manually, although I don't remember doing that myself. It's possible I did it accidentally while trying to fix something else. Is there a way to confirm if/when that index was deleted?

leandrojmp · June 12, 2024, 10:43pm

Not anymore, you would need to have the audit logs enabled before, and the audit logs also requires a paid license, not sure if you have as you didn't mention.

From what you shared, your policy would have not removed it, so it was deleted manually.

soad20000 · June 12, 2024, 10:55pm

Unfortunately I don't. Well nothing I can do about it I guess except be more careful. Thank you for your time.

soad20000 · June 28, 2024, 12:24am

So I finally found what the issue was. The hot, warm and cold phases were all set at 30 days and the delete phase at 365 days. BUT. The cold phase had the option enabled that says "Delete data after this phase". So after 30 days the data was getting deleted.

Silly mistake, I honestly can't remember setting it up that way but it makes sense that that was the issue. Will have to wait a while for the indexes to start rolling over to confirm that that was the issue but I'm pretty confident that's what it was.