# Issues with Data Retention - ELK Stack
**URL:** https://discuss.elastic.co/t/issues-with-data-retention-elk-stack/75055
**Category:** Elasticsearch
**Created:** [February 14, 2017, 2:44pm UTC](https://discuss.elastic.co/t/issues-with-data-retention-elk-stack/75055 "2017-02-14T14:44:45Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@dperkins](https://discuss.elastic.co/u/dperkins)
#### Post date: [February 14, 2017, 2:44pm UTC](https://discuss.elastic.co/t/issues-with-data-retention-elk-stack/75055/1 "2017-02-14T14:44:45Z")
Having an issue with a ELK stack not retaining data longer than 7 days. It's been running for over a year without any issue but when my gold license expired and I had to remove the Shield plugin as a result the cluster now only keeps 7 days worth of data. I doubt that this was related to that but I'm not sure where to look as this isn't normal behavior for Elasticsearch. Any help would be fantastic I can get whatever information is needed to try and narrow down the issue.
Thanks!
---
### Author:  [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)
#### Post date: [February 15, 2017, 6:14pm UTC](https://discuss.elastic.co/t/issues-with-data-retention-elk-stack/75055/2 "2017-02-15T18:14:34Z")
What version of Elasticsearch?
How are you maintaining your indices? Are you using Curator? Some custom script?
The only indices that are _ever_ auto-pruned are the Marvel (in x-pack, these are now called "monitoring-$PRODUCT", e.g. `monitoring-es-2-YYYY.MM.DD`) indices, which by default are kept at exactly 7 days. I am having a very hard time finding a correlation in any way to other data being deleted.
---
### Author:  [@dperkins](https://discuss.elastic.co/u/dperkins)
#### Post date: [February 15, 2017, 7:21pm UTC](https://discuss.elastic.co/t/issues-with-data-retention-elk-stack/75055/3 "2017-02-15T19:21:04Z")
Version - 2.3.0
We are currently only using one main index and before this started happening we weren't really maintaining it. It was just growing. Recently we had started to do XDELETE with a queried time frame to remove old data which I know isn't the best way of handling data.
---
### Author:  [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)
#### Post date: [February 15, 2017, 11:47pm UTC](https://discuss.elastic.co/t/issues-with-data-retention-elk-stack/75055/4 "2017-02-15T23:47:41Z")
Is there some cron job someone is running somewhere that is still doing that delete-by-query operation? I cannot fathom another way that deletes like this are happening.
---
### Author:  [@dperkins](https://discuss.elastic.co/u/dperkins)
#### Post date: [February 16, 2017, 4:36pm UTC](https://discuss.elastic.co/t/issues-with-data-retention-elk-stack/75055/5 "2017-02-16T16:36:11Z")
So there are no cron jobs running on the cluster at all. It feels like somehow the one api-log index we have is tied to the Marvel rotation. Would there be anyway of verifying that isn't happening?
---
### Author:  [@dperkins](https://discuss.elastic.co/u/dperkins)
#### Post date: [February 16, 2017, 9:33pm UTC](https://discuss.elastic.co/t/issues-with-data-retention-elk-stack/75055/6 "2017-02-16T21:33:14Z")
Aaron
It looks maybe we have a ttl set for the documents in the index but still trying to figure out where that is set. We changed the indices.ttl.interval and noticed that data stayed longer but would just be removed in larger chunks. We aren't adding a ttl to documents and as far as I can tell the index isn't imposing a ttl on the documents either.
---
### Author:  [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)
#### Post date: [February 16, 2017, 9:35pm UTC](https://discuss.elastic.co/t/issues-with-data-retention-elk-stack/75055/7 "2017-02-16T21:35:30Z")
A TTL would do that, too. Not the most efficient way to delete time-series data. I'd keep looking in templates and such to see if you can find the source.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 16, 2017, 9:35pm UTC](https://discuss.elastic.co/t/issues-with-data-retention-elk-stack/75055/8 "2017-03-16T21:35:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.