Issues with Data Retention - ELK Stack

Elastic Stack Elasticsearch
1

Having an issue with a ELK stack not retaining data longer than 7 days. It's been running for over a year without any issue but when my gold license expired and I had to remove the Shield plugin as a result the cluster now only keeps 7 days worth of data. I doubt that this was related to that but I'm not sure where to look as this isn't normal behavior for Elasticsearch. Any help would be fantastic I can get whatever information is needed to try and narrow down the issue.

Thanks!

2

What version of Elasticsearch?

How are you maintaining your indices? Are you using Curator? Some custom script?

The only indices that are ever auto-pruned are the Marvel (in x-pack, these are now called "monitoring-$PRODUCT", e.g. monitoring-es-2-YYYY.MM.DD) indices, which by default are kept at exactly 7 days. I am having a very hard time finding a correlation in any way to other data being deleted.

3

Version - 2.3.0
We are currently only using one main index and before this started happening we weren't really maintaining it. It was just growing. Recently we had started to do XDELETE with a queried time frame to remove old data which I know isn't the best way of handling data.

4

Is there some cron job someone is running somewhere that is still doing that delete-by-query operation? I cannot fathom another way that deletes like this are happening.

5

So there are no cron jobs running on the cluster at all. It feels like somehow the one api-log index we have is tied to the Marvel rotation. Would there be anyway of verifying that isn't happening?

6

Aaron
It looks maybe we have a ttl set for the documents in the index but still trying to figure out where that is set. We changed the indices.ttl.interval and noticed that data stayed longer but would just be removed in larger chunks. We aren't adding a ttl to documents and as far as I can tell the index isn't imposing a ttl on the documents either.

7

A TTL would do that, too. Not the most efficient way to delete time-series data. I'd keep looking in templates and such to see if you can find the source.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.