# Deleting documents by date

**URL:** <https://discuss.elastic.co/t/deleting-documents-by-date/82276>\
**Category:** Elasticsearch\
**Created:** [April 13, 2017, 10:51am UTC](https://discuss.elastic.co/t/deleting-documents-by-date/82276 "2017-04-13T10:51:23Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.elastic.co/u/kernelpanic)\
**Post date:** [April 13, 2017, 10:51am UTC](https://discuss.elastic.co/t/deleting-documents-by-date/82276/1 "2017-04-13T10:51:23Z")

</div>

Hello all, in my reading on this it does appear that the advice is to use time-based indices and delete the index rather than individual documents due to efficiency. Apart from efficiency concerns, are there any other reasons I shouldn't delete documents using a time range query instead of just deleting indices?

Thanks for any help.

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [April 13, 2017, 11:11am UTC](https://discuss.elastic.co/t/deleting-documents-by-date/82276/2 "2017-04-13T11:11:53Z")

</div>

It's a lot like comparing a sql `DELETE from TABLE where date<YYYY.MM.dd` to a sql `DROP TABLE`. One has potentially millions or billions of rows of atomic operations, where the other is a single operation. Which is more performant?

In terms of _operational_ efficiency, you end up needing a lot of extra read/write operations to actually delete things in a Lucene index. First, you have to _find_ what to delete (read operation). Then you need to _mark_ the documents for deletion (write operation). Then, at the next segment merge, Lucene has to _find_ the documents marked for deletion (read operation), and make new segments without those documents (write operation).

But it's not merely _operational_ efficiency, but segment fragmentation that will be your enemy over time, robbing you of valuable _storage_ efficiency. See [http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html](http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html) for a brilliant view into what happens with segment deletes (which come with document deletes) and how it interrupts the normal, efficient, tiered segments of a lucene index.

This is why we don't recommend delete\_by\_query as a solution for time-series data.

---

<div class="post-metadata">

**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.elastic.co/u/kernelpanic)\
**Post date:** [April 13, 2017, 11:21am UTC](https://discuss.elastic.co/t/deleting-documents-by-date/82276/3 "2017-04-13T11:21:18Z")

</div>

Thanks very much for getting back to me; we're just using the Elastic Stack for Windows event log collection with probably no more than 2 people running queries against it at any one time, so automatically deleting documents over night (as inefficient as it is) probably wouldn't be a concern in our situation. However as you've pointed out it may affect storage over time so I'll delete by indices instead, though I'm using a yearly index at the moment so I'll have to change the index pattern 🙂

Thanks for your help.

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [April 13, 2017, 11:41am UTC](https://discuss.elastic.co/t/deleting-documents-by-date/82276/4 "2017-04-13T11:41:03Z")

</div>

Why not use [Elasticsearch Curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html) (which has [2 Windows installation options](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/installation.html)) with the [Rollover](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/rollover.html) action? You can keep data for longer, and create fewer indices, by keeping rollover frequency low (usually by document count). Or rollover by age. It's up to you, but Curator can make an otherwise tedious index deletion ritual something you don't have to worry about as much.

---

<div class="post-metadata">

**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.elastic.co/u/kernelpanic)\
**Post date:** [April 13, 2017, 11:54am UTC](https://discuss.elastic.co/t/deleting-documents-by-date/82276/5 "2017-04-13T11:54:33Z")

</div>

Excellent - I did briefly look at Curator but didn't see the Rollover option; either way I'll use Curator to delete older data.

Thanks once again.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 11, 2017, 12:05pm UTC](https://discuss.elastic.co/t/deleting-documents-by-date/82276/6 "2017-05-11T12:05:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.