# How does ES handle deletes? (keeping a sliding window of documents) **URL:** \ **Category:** Elasticsearch\ **Created:** [October 3, 2012, 6:27am UTC](https://discuss.elastic.co/t/how-does-es-handle-deletes-keeping-a-sliding-window-of-documents/9219 "2012-10-03T06:27:58Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Theo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theo/32/2695_2.png) [@Theo](https://discuss.elastic.co/u/Theo)\ **Post date:** [October 3, 2012, 6:27am UTC](https://discuss.elastic.co/t/how-does-es-handle-deletes-keeping-a-sliding-window-of-documents/9219/1 "2012-10-03T06:27:58Z")
Hi, I'm evaluating ElasticSearch for a new thing I'm working on. What I haven't been able to figure out completely is how ES handles deletes. My plan is to continuously throw new documents into ES, but I only want to keep a sliding window of a couple of days, so every day I want to get rid of all the documents I added say a week ago. The documents will have a date so it doesn't look like there's going to be difficult to actually delete them, a delete-by query seems to do exactly what I want. However, how will ES handle that in terms of performance? What effects will it have on long time performance to always be deleting a big chunk of the data? Some databases would get heavily fragmented from that kind of operation, while others handle it gracefully by doing compactions from time to time. How does ES do it? As a test I added documents to ES until it used around a gigabyte of disk, then deleted a big chunk of the database and saw that eventually the size on disk decreased. That looks promising, but I'd like to be sure that what I saw was not due to something else. yours Theo -- ---
**Author:** ![Paul\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_smith/32/1323_2.png) [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)\ **Post date:** [October 3, 2012, 6:41am UTC](https://discuss.elastic.co/t/how-does-es-handle-deletes-keeping-a-sliding-window-of-documents/9219/2 "2012-10-03T06:41:03Z")
why not just have an index per day, and then create an alias across the per-day indices. Then you just delete the indices you don't need anymore. Basically like a 'truncate table' in SQL. you can easily query the REST api for all the indice names, and if they're named in a good format "myindex.2012-10-03' you can easily parse that and find a cull list to send Delete requests. On 3 October 2012 16:27, Theo [iconara@gmail.com](mailto:iconara@gmail.com) wrote: > Hi, > > I'm evaluating Elasticsearch for a new thing I'm working on. What I > haven't been able to figure out completely is how ES handles deletes. My > plan is to continuously throw new documents into ES, but I only want to > keep a sliding window of a couple of days, so every day I want to get rid > of all the documents I added say a week ago. The documents will have a date > so it doesn't look like there's going to be difficult to actually delete > them, a delete-by query seems to do exactly what I want. However, how will > ES handle that in terms of performance? What effects will it have on long > time performance to always be deleting a big chunk of the data? Some > databases would get heavily fragmented from that kind of operation, while > others handle it gracefully by doing compactions from time to time. How > does ES do it? > > As a test I added documents to ES until it used around a gigabyte of disk, > then deleted a big chunk of the database and saw that eventually the size > on disk decreased. That looks promising, but I'd like to be sure that what > I saw was not due to something else. > > yours > Theo > > -- -- ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [October 3, 2012, 6:41am UTC](https://discuss.elastic.co/t/how-does-es-handle-deletes-keeping-a-sliding-window-of-documents/9219/3 "2012-10-03T06:41:55Z")
You can use TTL field feature [http://www.elasticsearch.org/guide/reference/mapping/ttl-field.html](http://www.elasticsearch.org/guide/reference/mapping/ttl-field.html) Or you can create an index per week and add alias on this index. Let's say you create each monday an index named myindex201234 (2012 is the year and 34 the week number) and then you define an alias named myindex on myindex201234. The following week, you will add a new index myindex201235 and define myindex as alias. So, when you query for myindex you get all your docs. Deleting a week is only deleting an index which is really fast when comparing to deletebyquery. [http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html](http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html) HTH -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 3 oct. 2012 à 08:27, Theo [iconara@gmail.com](mailto:iconara@gmail.com) a écrit : Hi, I'm evaluating ElasticSearch for a new thing I'm working on. What I haven't been able to figure out completely is how ES handles deletes. My plan is to continuously throw new documents into ES, but I only want to keep a sliding window of a couple of days, so every day I want to get rid of all the documents I added say a week ago. The documents will have a date so it doesn't look like there's going to be difficult to actually delete them, a delete-by query seems to do exactly what I want. However, how will ES handle that in terms of performance? What effects will it have on long time performance to always be deleting a big chunk of the data? Some databases would get heavily fragmented from that kind of operation, while others handle it gracefully by doing compactions from time to time. How does ES do it? As a test I added documents to ES until it used around a gigabyte of disk, then deleted a big chunk of the database and saw that eventually the size on disk decreased. That looks promising, but I'd like to be sure that what I saw was not due to something else. ## yours Theo -- ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [October 3, 2012, 6:42am UTC](https://discuss.elastic.co/t/how-does-es-handle-deletes-keeping-a-sliding-window-of-documents/9219/4 "2012-10-03T06:42:36Z")
Paul was faster than me 😉 -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 3 oct. 2012 à 08:41, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) a écrit : why not just have an index per day, and then create an alias across the per-day indices. Then you just delete the indices you don't need anymore. Basically like a 'truncate table' in SQL. you can easily query the REST api for all the indice names, and if they're named in a good format "myindex.2012-10-03' you can easily parse that and find a cull list to send Delete requests. On 3 October 2012 16:27, Theo [iconara@gmail.com](mailto:iconara@gmail.com) wrote: > Hi, > > I'm evaluating Elasticsearch for a new thing I'm working on. What I haven't been able to figure out completely is how ES handles deletes. My plan is to continuously throw new documents into ES, but I only want to keep a sliding window of a couple of days, so every day I want to get rid of all the documents I added say a week ago. The documents will have a date so it doesn't look like there's going to be difficult to actually delete them, a delete-by query seems to do exactly what I want. However, how will ES handle that in terms of performance? What effects will it have on long time performance to always be deleting a big chunk of the data? Some databases would get heavily fragmented from that kind of operation, while others handle it gracefully by doing compactions from time to time. How does ES do it? > > As a test I added documents to ES until it used around a gigabyte of disk, then deleted a big chunk of the database and saw that eventually the size on disk decreased. That looks promising, but I'd like to be sure that what I saw was not due to something else. > > ## yours Theo -- -- ---
**Author:** ![Theo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theo/32/2695_2.png) [@Theo](https://discuss.elastic.co/u/Theo)\ **Post date:** [October 3, 2012, 7:06am UTC](https://discuss.elastic.co/t/how-does-es-handle-deletes-keeping-a-sliding-window-of-documents/9219/5 "2012-10-03T07:06:13Z")
Thanks! The idea of keeping an index per day or week makes sense. We've done something like that in other databases when we've had to keep sliding windows, but it's always a hassle because you have to add a layer that maps your queries against the right database depending on the date in the query and so on. Index aliases seem to solve that very nicely -- I wasn't aware of them but now I'll go read all about it. T# On Wednesday, October 3, 2012 8:42:45 AM UTC+2, David Pilato wrote: > Paul was faster than me 😉 > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 3 oct. 2012 à 08:41, Paul Smith \<[tallp...@gmail.com](mailto:tallp...@gmail.com) \\> a > écrit : > > why not just have an index per day, and then create an alias across the > per-day indices. Then you just delete the indices you don't need anymore. > Basically like a 'truncate table' in SQL. you can easily query the REST > api for all the indice names, and if they're named in a good format > "myindex.2012-10-03' you can easily parse that and find a cull list to send > Delete requests. > > On 3 October 2012 16:27, Theo \<[ico...@gmail.com](mailto:ico...@gmail.com) \\> wrote: > > > Hi, > > > > I'm evaluating Elasticsearch for a new thing I'm working on. What I > > haven't been able to figure out completely is how ES handles deletes. My > > plan is to continuously throw new documents into ES, but I only want to > > keep a sliding window of a couple of days, so every day I want to get rid > > of all the documents I added say a week ago. The documents will have a date > > so it doesn't look like there's going to be difficult to actually delete > > them, a delete-by query seems to do exactly what I want. However, how will > > ES handle that in terms of performance? What effects will it have on long > > time performance to always be deleting a big chunk of the data? Some > > databases would get heavily fragmented from that kind of operation, while > > others handle it gracefully by doing compactions from time to time. How > > does ES do it? > > > > As a test I added documents to ES until it used around a gigabyte of > > disk, then deleted a big chunk of the database and saw that eventually the > > size on disk decreased. That looks promising, but I'd like to be sure that > > what I saw was not due to something else. > > > > yours > > Theo > > > > -- > > -- -- ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [October 3, 2012, 7:20am UTC](https://discuss.elastic.co/t/how-does-es-handle-deletes-keeping-a-sliding-window-of-documents/9219/6 "2012-10-03T07:20:08Z")
Check also this presentation > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. Shay mentioned this sliding window ("time" data flow) strategy in his presentation. Regards, Lukas On Wed, Oct 3, 2012 at 9:06 AM, Theo [iconara@gmail.com](mailto:iconara@gmail.com) wrote: > Thanks! The idea of keeping an index per day or week makes sense. We've > done something like that in other databases when we've had to keep sliding > windows, but it's always a hassle because you have to add a layer that maps > your queries against the right database depending on the date in the query > and so on. Index aliases seem to solve that very nicely -- I wasn't aware > of them but now I'll go read all about it. > > T# > > On Wednesday, October 3, 2012 8:42:45 AM UTC+2, David Pilato wrote: > > > Paul was faster than me 😉 > > > > -- > > David 😉 > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > Le 3 oct. 2012 à 08:41, Paul Smith [tallp...@gmail.com](mailto:tallp...@gmail.com) a écrit : > > > > why not just have an index per day, and then create an alias across the > > per-day indices. Then you just delete the indices you don't need anymore. > > Basically like a 'truncate table' in SQL. you can easily query the REST > > api for all the indice names, and if they're named in a good format > > "myindex.2012-10-03' you can easily parse that and find a cull list to send > > Delete requests. > > > > On 3 October 2012 16:27, Theo [ico...@gmail.com](mailto:ico...@gmail.com) wrote: > > > > > Hi, > > > > > > I'm evaluating Elasticsearch for a new thing I'm working on. What I > > > haven't been able to figure out completely is how ES handles deletes. My > > > plan is to continuously throw new documents into ES, but I only want to > > > keep a sliding window of a couple of days, so every day I want to get rid > > > of all the documents I added say a week ago. The documents will have a date > > > so it doesn't look like there's going to be difficult to actually delete > > > them, a delete-by query seems to do exactly what I want. However, how will > > > ES handle that in terms of performance? What effects will it have on long > > > time performance to always be deleting a big chunk of the data? Some > > > databases would get heavily fragmented from that kind of operation, while > > > others handle it gracefully by doing compactions from time to time. How > > > does ES do it? > > > > > > As a test I added documents to ES until it used around a gigabyte of > > > disk, then deleted a big chunk of the database and saw that eventually the > > > size on disk decreased. That looks promising, but I'd like to be sure that > > > what I saw was not due to something else. > > > > > > yours > > > Theo > > > > > > -- > > > > -- > > -- -- ---
**Author:** ![Raffaele\_Sena](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raffaele_sena/32/1759_2.png) [@Raffaele\_Sena](https://discuss.elastic.co/u/Raffaele_Sena)\ **Post date:** [October 3, 2012, 10:42pm UTC](https://discuss.elastic.co/t/how-does-es-handle-deletes-keeping-a-sliding-window-of-documents/9219/7 "2012-10-03T22:42:22Z")
I just realized ES also support a TTL on documents ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/index_.html) look for TTL). Unless you really want to delete old documents only when you add new ones, this may be what you are looking for. -- Raffaele On Wed, Oct 3, 2012 at 12:20 AM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote: > Check also this presentation > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/videos/2012/06/05/big-data-search-and-analytics.html) > > Shay mentioned this sliding window ("time" data flow) strategy in his > presentation. > > Regards, > Lukas > > On Wed, Oct 3, 2012 at 9:06 AM, Theo [iconara@gmail.com](mailto:iconara@gmail.com) wrote: > > > Thanks! The idea of keeping an index per day or week makes sense. We've > > done something like that in other databases when we've had to keep sliding > > windows, but it's always a hassle because you have to add a layer that maps > > your queries against the right database depending on the date in the query > > and so on. Index aliases seem to solve that very nicely -- I wasn't aware of > > them but now I'll go read all about it. > > > > T# > > > > On Wednesday, October 3, 2012 8:42:45 AM UTC+2, David Pilato wrote: > > > > > Paul was faster than me 😉 > > > > > > -- > > > David 😉 > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > Le 3 oct. 2012 à 08:41, Paul Smith [tallp...@gmail.com](mailto:tallp...@gmail.com) a écrit : > > > > > > why not just have an index per day, and then create an alias across the > > > per-day indices. Then you just delete the indices you don't need anymore. > > > Basically like a 'truncate table' in SQL. you can easily query the REST api > > > for all the indice names, and if they're named in a good format > > > "myindex.2012-10-03' you can easily parse that and find a cull list to send > > > Delete requests. > > > > > > On 3 October 2012 16:27, Theo [ico...@gmail.com](mailto:ico...@gmail.com) wrote: > > > > > > > Hi, > > > > > > > > I'm evaluating Elasticsearch for a new thing I'm working on. What I > > > > haven't been able to figure out completely is how ES handles deletes. My > > > > plan is to continuously throw new documents into ES, but I only want to keep > > > > a sliding window of a couple of days, so every day I want to get rid of all > > > > the documents I added say a week ago. The documents will have a date so it > > > > doesn't look like there's going to be difficult to actually delete them, a > > > > delete-by query seems to do exactly what I want. However, how will ES handle > > > > that in terms of performance? What effects will it have on long time > > > > performance to always be deleting a big chunk of the data? Some databases > > > > would get heavily fragmented from that kind of operation, while others > > > > handle it gracefully by doing compactions from time to time. How does ES do > > > > it? > > > > > > > > As a test I added documents to ES until it used around a gigabyte of > > > > disk, then deleted a big chunk of the database and saw that eventually the > > > > size on disk decreased. That looks promising, but I'd like to be sure that > > > > what I saw was not due to something else. > > > > > > > > yours > > > > Theo > > > > > > > > -- > > > > > > -- > > > > -- > > -- -- ---
**Author:** ![Theo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theo/32/2695_2.png) [@Theo](https://discuss.elastic.co/u/Theo)\ **Post date:** [October 4, 2012, 6:17am UTC](https://discuss.elastic.co/t/how-does-es-handle-deletes-keeping-a-sliding-window-of-documents/9219/8 "2012-10-04T06:17:41Z")
TTL is very convenient, but it comes back to my original question: how does the underlying database handle deletions of lots of documents? will it become fragmented over time and get slower? this happens for example in MongoDB or MySQL/InnoDB if you have a high insertion and deletion rate. Cassandra does not suffer from this effect because it continuously compacts its on-disk data. how does ES perform a delete? does it create a hole in the on-disk datastructures that will later be filled by new data, or does it add a tombstone that will get compacted away at a later time, or does it do something else? (I guess the question might actually be how Lucene does this). I think that the one-index-per-week solution will work great, but if ES handles deletions gracefully without getting fragmented that would be very good too. T# On Thursday, October 4, 2012 12:42:26 AM UTC+2, Raffaele Sena wrote: > I just realized ES also support a TTL on documents > ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/index_.html) look for > TTL). > > Unless you really want to delete old documents only when you add new > ones, this may be what you are looking for. > > -- Raffaele > > On Wed, Oct 3, 2012 at 12:20 AM, Lukáš Vlček \<[lukas...@gmail.com](mailto:lukas...@gmail.com)\\> > wrote: > > > Check also this presentation > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/videos/2012/06/05/big-data-search-and-analytics.html) > > > Shay mentioned this sliding window ("time" data flow) strategy in his > > presentation. > > > > Regards, > > Lukas > > > > On Wed, Oct 3, 2012 at 9:06 AM, Theo \<[ico...@gmail.com](mailto:ico...@gmail.com) \\> > > wrote: > > > > > Thanks! The idea of keeping an index per day or week makes sense. We've > > > done something like that in other databases when we've had to keep > > > sliding > > > windows, but it's always a hassle because you have to add a layer that > > > maps > > > your queries against the right database depending on the date in the > > > query > > > and so on. Index aliases seem to solve that very nicely -- I wasn't > > > aware of > > > them but now I'll go read all about it. > > > > > > T# > > > > > > On Wednesday, October 3, 2012 8:42:45 AM UTC+2, David Pilato wrote: > > > > > > > Paul was faster than me 😉 > > > > > > > > -- > > > > David 😉 > > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > > > Le 3 oct. 2012 à 08:41, Paul Smith [tallp...@gmail.com](mailto:tallp...@gmail.com) a écrit : > > > > > > > > why not just have an index per day, and then create an alias across > > > > the > > > > per-day indices. Then you just delete the indices you don't need > > > > anymore. > > > > Basically like a 'truncate table' in SQL. you can easily query the > > > > REST api > > > > for all the indice names, and if they're named in a good format > > > > "myindex.2012-10-03' you can easily parse that and find a cull list to > > > > send > > > > Delete requests. > > > > > > > > On 3 October 2012 16:27, Theo [ico...@gmail.com](mailto:ico...@gmail.com) wrote: > > > > > > > > > Hi, > > > > > > > > > > I'm evaluating Elasticsearch for a new thing I'm working on. What I > > > > > haven't been able to figure out completely is how ES handles deletes. > > > > > My > > > > > plan is to continuously throw new documents into ES, but I only want > > > > > to keep > > > > > a sliding window of a couple of days, so every day I want to get rid > > > > > of all > > > > > the documents I added say a week ago. The documents will have a date > > > > > so it > > > > > doesn't look like there's going to be difficult to actually delete > > > > > them, a > > > > > delete-by query seems to do exactly what I want. However, how will ES > > > > > handle > > > > > that in terms of performance? What effects will it have on long time > > > > > performance to always be deleting a big chunk of the data? Some > > > > > databases > > > > > would get heavily fragmented from that kind of operation, while > > > > > others > > > > > handle it gracefully by doing compactions from time to time. How does > > > > > ES do > > > > > it? > > > > > > > > > > As a test I added documents to ES until it used around a gigabyte of > > > > > disk, then deleted a big chunk of the database and saw that > > > > > eventually the > > > > > size on disk decreased. That looks promising, but I'd like to be sure > > > > > that > > > > > what I saw was not due to something else. > > > > > > > > > > yours > > > > > Theo > > > > > > > > > > -- > > > > > > > > -- > > > > > > -- > > > > -- -- ---
**Author:** ![Paul\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_smith/32/1323_2.png) [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)\ **Post date:** [October 4, 2012, 7:01am UTC](https://discuss.elastic.co/t/how-does-es-handle-deletes-keeping-a-sliding-window-of-documents/9219/9 "2012-10-04T07:01:00Z")
On 4 October 2012 16:17, Theo [iconara@gmail.com](mailto:iconara@gmail.com) wrote: > TTL is very convenient, but it comes back to my original question: how > does the underlying database handle deletions of lots of documents? will it > become fragmented over time and get slower? this happens for example in > MongoDB or MySQL/InnoDB if you have a high insertion and deletion rate. > Cassandra does not suffer from this effect because it continuously compacts > its on-disk data. how does ES perform a delete? does it create a hole in > the on-disk datastructures that will later be filled by new data, or does > it add a tombstone that will get compacted away at a later time, or does it > do something else? (I guess the question might actually be how Lucene does > this). > > I think that the one-index-per-week solution will work great, but if ES > handles deletions gracefully without getting fragmented that would be very > good too. As I understand how Lucene works, a delete marks a specific Document ID (a Lucene internal term) as deleted. Each segment maintains a delete marker (\*.del files for each segment in the indices directory of a shard). This makes it a fast delete, because no real index modification needs to be made other than writing the ID to be deleted (that file is mapped into memory too). When a search happens, all matching Doc ID's form part of a matching bitmask for the matches, which then get the deletes applied to them by masking out the document Ids of 'matches' that are no longer there. Obviously in a newly created index with no mutations, the delete file is empty, so this is a no-op, so there's a slight overhead when theres deletes to be applied. Since an 'update' in Lucene is really a delete and and add operation, these deletes accumulate over time, but as new items are added, merges are triggered so that the # and size of segment files doesn't get too crazy. During the merge process the deleted records are wiped out during the merge compaction. So wholesale deletes by wiping out large segments of your document space will end up with lots of items in the individual segment delete files, which isn't _terrible_, just not optimal. There is potential depending on the distribution of your documents and their contents on how the term frequencies etc are, that the actual searches may be slower because there's all these redundant document Ids in each terms vector (basically wasted space) that end up getting nulled out, so you do pay a penalty for them, but it's not a massive overhead. One can optimize after you bulk delete of course by using the \_optimize API call to compact/merge segments together to remove all these, but it's a lot of IO for large indices if you've blown a way a huge chunk, and way slower than a file delete that ends up happening when you \_delete the specific indice. I like to think of it as the difference between an SQL "DELETE" and "TRUNCATE", the former being 'logged for recovery' (ala in the .del file, not _realyl_ recoverable, I'm not sure what happens if you manually remove that .del file) while the latter is much faster. So, deleting an index wholesale is way more efficient in Lucene world if you can design your indices & aliases that way, but searches are still pretty good if you do block deletes within an index, just not _as_ good as if the index is in a clean state, you'll waste memory and disk head seeks over time, and you want searching to be fast right? That's why you picked or are looking at Elasticsearch! 🙂 hopefully that helps Paul -- ---
**Author:** ![Theo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theo/32/2695_2.png) [@Theo](https://discuss.elastic.co/u/Theo)\ **Post date:** [October 4, 2012, 7:06am UTC](https://discuss.elastic.co/t/how-does-es-handle-deletes-keeping-a-sliding-window-of-documents/9219/10 "2012-10-04T07:06:34Z")
Thanks Paul, that was exactly the explanation I was looking for. Now I feel I have a much better understanding of the situation. For my particular use case I think partitioning is the way to go (i.e. a new index per week/day/month/whatever), but it's great to know how ES would handle TTL and individual deletes, and that it would probably work that way too, just not as efficiently. T# On Thursday, October 4, 2012 9:01:03 AM UTC+2, tallpsmith wrote: > On 4 October 2012 16:17, Theo \<[ico...@gmail.com](mailto:ico...@gmail.com) \\> wrote: > > > TTL is very convenient, but it comes back to my original question: how > > does the underlying database handle deletions of lots of documents? will it > > become fragmented over time and get slower? this happens for example in > > MongoDB or MySQL/InnoDB if you have a high insertion and deletion rate. > > Cassandra does not suffer from this effect because it continuously compacts > > its on-disk data. how does ES perform a delete? does it create a hole in > > the on-disk datastructures that will later be filled by new data, or does > > it add a tombstone that will get compacted away at a later time, or does it > > do something else? (I guess the question might actually be how Lucene does > > this). > > > > I think that the one-index-per-week solution will work great, but if ES > > handles deletions gracefully without getting fragmented that would be very > > good too. > > As I understand how Lucene works, a delete marks a specific Document ID (a > Lucene internal term) as deleted. Each segment maintains a delete marker > (\*.del files for each segment in the indices directory of a shard). This > makes it a fast delete, because no real index modification needs to be made > other than writing the ID to be deleted (that file is mapped into memory > too). When a search happens, all matching Doc ID's form part of a matching > bitmask for the matches, which then get the deletes applied to them by > masking out the document Ids of 'matches' that are no longer there. > Obviously in a newly created index with no mutations, the delete file is > empty, so this is a no-op, so there's a slight overhead when theres deletes > to be applied. > > Since an 'update' in Lucene is really a delete and and add operation, > these deletes accumulate over time, but as new items are added, merges are > triggered so that the # and size of segment files doesn't get too crazy. > During the merge process the deleted records are wiped out during the > merge compaction. > > So wholesale deletes by wiping out large segments of your document space > will end up with lots of items in the individual segment delete files, > which isn't _terrible_, just not optimal. There is potential depending on > the distribution of your documents and their contents on how the term > frequencies etc are, that the actual searches may be slower because there's > all these redundant document Ids in each terms vector (basically wasted > space) that end up getting nulled out, so you do pay a penalty for them, > but it's not a massive overhead. > > One can optimize after you bulk delete of course by using the \_optimize > API call to compact/merge segments together to remove all these, but it's a > lot of IO for large indices if you've blown a way a huge chunk, and way > slower than a file delete that ends up happening when you \_delete the > specific indice. I like to think of it as the difference between an SQL > "DELETE" and "TRUNCATE", the former being 'logged for recovery' (ala in the > .del file, not _realyl_ recoverable, I'm not sure what happens if you > manually remove that .del file) while the latter is much faster. > > So, deleting an index wholesale is way more efficient in Lucene world if > you can design your indices & aliases that way, but searches are still > pretty good if you do block deletes within an index, just not _as_ good as > if the index is in a clean state, you'll waste memory and disk head seeks > over time, and you want searching to be fast right? That's why you picked > or are looking at Elasticsearch! 🙂 > > hopefully that helps > > Paul -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:10am UTC](https://discuss.elastic.co/t/how-does-es-handle-deletes-keeping-a-sliding-window-of-documents/9219/11 "2017-07-06T03:10:14Z")