# Automatic deleting the documents from indices **URL:** \ **Category:** Elasticsearch\ **Created:** [May 8, 2019, 5:16am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097 "2019-05-08T05:16:47Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Vikash\_Singh1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikash_singh1/32/42119_2.png) [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Post date:** [May 8, 2019, 5:16am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/1 "2019-05-08T05:16:47Z")
Hi there!! Since I have started using **ELK** it make me more curious hence I keep on exploring and hence the doubts also increases rapidly. The biggest challenge in analysis is the size, Now I have an question is there anyway through which older documents from indices automatically gets deleted. As I have managed to discover the mechanism where the new documents are automatically added to my indices hence automatically deletion of documents from indices would be very helpful. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 8, 2019, 5:46am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/2 "2019-05-08T05:46:11Z")
What sort of use case are you using the stack for? ---
**Author:** ![Vikash\_Singh1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikash_singh1/32/42119_2.png) [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Post date:** [May 8, 2019, 6:40am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/3 "2019-05-08T06:40:05Z")
Currently I am using it for Logging and Log Analysis, Full text search along with visualizing. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 8, 2019, 6:41am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/4 "2019-05-08T06:41:03Z")
Are you using time based indices? ---
**Author:** ![Vikash\_Singh1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikash_singh1/32/42119_2.png) [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Post date:** [May 8, 2019, 6:44am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/5 "2019-05-08T06:44:14Z")
Yes I am using time base indices ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 8, 2019, 6:46am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/6 "2019-05-08T06:46:04Z")
Then just delete the index instead of the document. Take a look at ILM as well, it might be better. ---
**Author:** ![Vikash\_Singh1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikash_singh1/32/42119_2.png) [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Post date:** [May 8, 2019, 6:52am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/7 "2019-05-08T06:52:01Z")
I am using pipelines.yml so that whenever I have an input it will dynamically updates those file in an index. But space is the real issues because we can't have unlimited space and you know these logs size they are growing rapidly eating all the space. So I thought what if I can automatically delete the old documents from my index this way no one has to be worried about the space. Previously I was checking the curator but it will delete the whole index... Then I tried the delete api where certain documents can be deleted but manually.. ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [May 8, 2019, 11:34am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/8 "2019-05-08T11:34:47Z")
For time series data, you should use rollover indices, and delete the entire index, still. ILM is a great way to do this automatically based on index/shard size constraints. If you still need to delete indices according to space constraints, then you could still fall back and use Curator instead of ILM, but you'd be responsible to have Curator run periodically, using `cron` or something similar. ---
**Author:** ![Vikash\_Singh1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikash_singh1/32/42119_2.png) [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Post date:** [May 9, 2019, 4:24am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/9 "2019-05-09T04:24:47Z")
Hence there is no way through which I can periodically delete the old documents from time based indices?? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 9, 2019, 5:19am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/10 "2019-05-09T05:19:34Z")
The whole point of having time-based indices is that you can delete complete indices which is much more efficient than deleting documents individually from an index. ---
**Author:** ![Vikash\_Singh1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikash_singh1/32/42119_2.png) [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Post date:** [May 9, 2019, 5:20am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/11 "2019-05-09T05:20:33Z")
But if we delete the index our visualization will also be deleted. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 9, 2019, 5:23am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/12 "2019-05-09T05:23:03Z")
An index pattern typically matches a set of time based indices so deleting the oldest index should be the same as deleting the data and visualisations should not be affected. ---
**Author:** ![Vikash\_Singh1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikash_singh1/32/42119_2.png) [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Post date:** [May 9, 2019, 5:28am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/13 "2019-05-09T05:28:21Z")
Naah actually I deleted an index and all the visualizations related to that index were deleted too. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 9, 2019, 5:36am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/14 "2019-05-09T05:36:09Z")
All Kibana visualisations are kept in a separate index so that should not be the case. Can you please describe your setup and exactly what you did? ---
**Author:** ![Vikash\_Singh1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikash_singh1/32/42119_2.png) [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Post date:** [May 9, 2019, 5:42am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/15 "2019-05-09T05:42:57Z")
OK...few more questions 1. I am importing logs via logstash using pipelines.yml. Once my index is created after that I can dynamically update my index (I ain't closing the logstash) just by uploading the logs in specified location. Now once my index is deleted and afterwards if i add some new logs in that location will it automatically create a new one? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 9, 2019, 5:44am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/16 "2019-05-09T05:44:55Z")
What does your Logstash config look like? How are you creating indices? Time-based indices are typically used with immutable data which makes me wonder if you really are using this. ---
**Author:** ![Vikash\_Singh1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikash_singh1/32/42119_2.png) [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Post date:** [May 9, 2019, 5:46am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/17 "2019-05-09T05:46:08Z")
Shall I paste pipelines.yml? or you need to see the logstash.conf file? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 9, 2019, 5:46am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/18 "2019-05-09T05:46:47Z")
I am interseted in the Elasticsearch output section of the conf files. ---
**Author:** ![Vikash\_Singh1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikash_singh1/32/42119_2.png) [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Post date:** [May 9, 2019, 5:47am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/19 "2019-05-09T05:47:40Z")
This is apache.conf file: input { file { path =\> "/elk/Weblog/\*" start\_position =\> "beginning" } } filter { grok { match =\> { "message" =\> "%{COMBINEDAPACHELOG}"} } geoip { source =\> "clientip" } } output { elasticsearch { hosts =\> ["10.11.109.7:9200"] index =\> "logstash\_apchelogs" } } ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 9, 2019, 5:53am UTC](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097/20 "2019-05-09T05:53:19Z")
If you are specifying a single index name you are not using time-based indices, which is what Logstash creates by default, e.g. `logstash-2019-05-09`. In that case deleting the index will delete all data. In such scenarion you probably need to use delete-by-query to delete documents, but be aware this is much more expensove than deleting a complete time-based index. [Next page](https://discuss.elastic.co/t/automatic-deleting-the-documents-from-indices/180097.md?page=2)