# Mass delete by query **URL:** \ **Category:** Elasticsearch\ **Created:** [March 31, 2014, 9:50pm UTC](https://discuss.elastic.co/t/mass-delete-by-query/16727 "2014-03-31T21:50:14Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![slushi](https://avatars.discourse-cdn.com/v4/letter/s/3e96dc/32.png) [@slushi](https://discuss.elastic.co/u/slushi)\ **Post date:** [March 31, 2014, 9:50pm UTC](https://discuss.elastic.co/t/mass-delete-by-query/16727/1 "2014-03-31T21:50:14Z")
I have varying data retention requirements I am trying to balance (I am continuously indexing new documents): - 1% of my documents need to be kept forever - 10% need to be kept 1 year - the remainder needs to be kept for 1 month I can easily set properties indicating the retention policy for each document and then periodically do a "delete by query". However, since the delete would remove 89% of the indexed documents, would there be any potential performance problems with this straightforward approach? I guess this is a YMMV type thing, but I was just wondering what the typical approach is here. Would it be necessary to perhaps filter the query to not affect so many documents at once? Would query performance be greatly impacted? The alternate approach I was thinking would be to create separate indices for each retention type. Cleanup would be easier, but unfortunately a document's retention policy can be upgraded/downgraded so that could be a little messy to keep consistent. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/672e4d70-b9f9-4f6c-b22e-4287ef5a27ab%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/672e4d70-b9f9-4f6c-b22e-4287ef5a27ab%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Kevin\_Wang](https://avatars.discourse-cdn.com/v4/letter/k/f6c823/32.png) [@Kevin\_Wang](https://discuss.elastic.co/u/Kevin_Wang)\ **Post date:** [March 31, 2014, 9:52pm UTC](https://discuss.elastic.co/t/mass-delete-by-query/16727/2 "2014-03-31T21:52:28Z")
Why not use TTL for document? [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html) On Tuesday, April 1, 2014 8:50:14 AM UTC+11, slushi wrote: > I have varying data retention requirements I am trying to balance (I am > continuously indexing new documents): > > - 1% of my documents need to be kept forever > - 10% need to be kept 1 year > - the remainder needs to be kept for 1 month > > I can easily set properties indicating the retention policy for each > document and then periodically do a "delete by query". However, since the > delete would remove 89% of the indexed documents, would there be any > potential performance problems with this straightforward approach? I guess > this is a YMMV type thing, but I was just wondering what the typical > approach is here. Would it be necessary to perhaps filter the query to not > affect so many documents at once? Would query performance be greatly > impacted? > > The alternate approach I was thinking would be to create separate indices > for each retention type. Cleanup would be easier, but unfortunately a > document's retention policy can be upgraded/downgraded so that could be a > little messy to keep consistent. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/eefba11c-d147-4e02-b84b-bc8f90a08e3f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/eefba11c-d147-4e02-b84b-bc8f90a08e3f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![slushi](https://avatars.discourse-cdn.com/v4/letter/s/3e96dc/32.png) [@slushi](https://discuss.elastic.co/u/slushi)\ **Post date:** [March 31, 2014, 10:00pm UTC](https://discuss.elastic.co/t/mass-delete-by-query/16727/3 "2014-03-31T22:00:16Z")
I attended an Elasticsearch meet up and at some point it was mentioned that TTL use is discouraged, but yes this would make a lot of sense here. Also the 1 year thing is really a guesstimate, we want to keep as much of that data as possible. I guess maybe with TTL you may not have as much control when the document deletion and possible segment merging? I am not that familiar with Elasticsearch performance stuff yet (we just started looking into using ES). On Monday, March 31, 2014 5:52:28 PM UTC-4, Kevin Wang wrote: > Why not use TTL for document? > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html) > > On Tuesday, April 1, 2014 8:50:14 AM UTC+11, slushi wrote: > > > I have varying data retention requirements I am trying to balance (I am > > continuously indexing new documents): > > > > - 1% of my documents need to be kept forever > > - 10% need to be kept 1 year > > - the remainder needs to be kept for 1 month > > > > I can easily set properties indicating the retention policy for each > > document and then periodically do a "delete by query". However, since the > > delete would remove 89% of the indexed documents, would there be any > > potential performance problems with this straightforward approach? I guess > > this is a YMMV type thing, but I was just wondering what the typical > > approach is here. Would it be necessary to perhaps filter the query to not > > affect so many documents at once? Would query performance be greatly > > impacted? > > > > The alternate approach I was thinking would be to create separate indices > > for each retention type. Cleanup would be easier, but unfortunately a > > document's retention policy can be upgraded/downgraded so that could be a > > little messy to keep consistent. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9b685cff-e956-473a-935e-9546b2ea59b3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9b685cff-e956-473a-935e-9546b2ea59b3%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 1, 2014, 5:58am UTC](https://discuss.elastic.co/t/mass-delete-by-query/16727/4 "2014-04-01T05:58:04Z")
If you know in advance which doc should be removed (i mean at index time), you should send the document to an index which should be entirely removed after a given period. Makes sense? -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 1 avr. 2014 à 00:00, slushi [kireetreddy@gmail.com](mailto:kireetreddy@gmail.com) a écrit : I attended an Elasticsearch meet up and at some point it was mentioned that TTL use is discouraged, but yes this would make a lot of sense here. Also the 1 year thing is really a guesstimate, we want to keep as much of that data as possible. I guess maybe with TTL you may not have as much control when the document deletion and possible segment merging? I am not that familiar with Elasticsearch performance stuff yet (we just started looking into using ES). > On Monday, March 31, 2014 5:52:28 PM UTC-4, Kevin Wang wrote: > Why not use TTL for document? [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html) > > > On Tuesday, April 1, 2014 8:50:14 AM UTC+11, slushi wrote: > > I have varying data retention requirements I am trying to balance (I am continuously indexing new documents): > > 1% of my documents need to be kept forever > > 10% need to be kept 1 year > > the remainder needs to be kept for 1 month > > I can easily set properties indicating the retention policy for each document and then periodically do a "delete by query". However, since the delete would remove 89% of the indexed documents, would there be any potential performance problems with this straightforward approach? I guess this is a YMMV type thing, but I was just wondering what the typical approach is here. Would it be necessary to perhaps filter the query to not affect so many documents at once? Would query performance be greatly impacted? > > > > The alternate approach I was thinking would be to create separate indices for each retention type. Cleanup would be easier, but unfortunately a document's retention policy can be upgraded/downgraded so that could be a little messy to keep consistent. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9b685cff-e956-473a-935e-9546b2ea59b3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9b685cff-e956-473a-935e-9546b2ea59b3%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CF5F858B-18DF-499F-96C2-F21AE4CD13EC%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/CF5F858B-18DF-499F-96C2-F21AE4CD13EC%40pilato.fr). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![slushi](https://avatars.discourse-cdn.com/v4/letter/s/3e96dc/32.png) [@slushi](https://discuss.elastic.co/u/slushi)\ **Post date:** [April 1, 2014, 6:03am UTC](https://discuss.elastic.co/t/mass-delete-by-query/16727/5 "2014-04-01T06:03:12Z")
yes, unfortunately it’s not completely known at index time. I would need to keep the separate indices in sync when a retention policy change occurs. attempting this seems like it could open up a whole can of worms. On Tuesday, April 1, 2014 1:58:04 AM UTC-4, David Pilato wrote: > If you know in advance which doc should be removed (i mean at index time), > you should send the document to an index which should be entirely removed > after a given period. > > Makes sense? > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 1 avr. 2014 à 00:00, slushi \<[kiree...@gmail.com](mailto:kiree...@gmail.com) \\> a > écrit : > > I attended an Elasticsearch meet up and at some point it was mentioned > that TTL use is discouraged, but yes this would make a lot of sense here. > Also the 1 year thing is really a guesstimate, we want to keep as much of > that data as possible. I guess maybe with TTL you may not have as much > control when the document deletion and possible segment merging? I am not > that familiar with Elasticsearch performance stuff yet (we just started > looking into using ES). > > On Monday, March 31, 2014 5:52:28 PM UTC-4, Kevin Wang wrote: > > > Why not use TTL for document? > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html) > > > > On Tuesday, April 1, 2014 8:50:14 AM UTC+11, slushi wrote: > > > > > I have varying data retention requirements I am trying to balance (I am > > > continuously indexing new documents): > > > > > > - 1% of my documents need to be kept forever > > > - 10% need to be kept 1 year > > > - the remainder needs to be kept for 1 month > > > > > > I can easily set properties indicating the retention policy for each > > > document and then periodically do a "delete by query". However, since the > > > delete would remove 89% of the indexed documents, would there be any > > > potential performance problems with this straightforward approach? I guess > > > this is a YMMV type thing, but I was just wondering what the typical > > > approach is here. Would it be necessary to perhaps filter the query to not > > > affect so many documents at once? Would query performance be greatly > > > impacted? > > > > > > The alternate approach I was thinking would be to create separate > > > indices for each retention type. Cleanup would be easier, but unfortunately > > > a document's retention policy can be upgraded/downgraded so that could be a > > > little messy to keep consistent. > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/9b685cff-e956-473a-935e-9546b2ea59b3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9b685cff-e956-473a-935e-9546b2ea59b3%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/9b685cff-e956-473a-935e-9546b2ea59b3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9b685cff-e956-473a-935e-9546b2ea59b3%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/eec089d7-0cef-4a9b-b53f-7dce55ad2bfd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/eec089d7-0cef-4a9b-b53f-7dce55ad2bfd%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 1, 2014, 6:18am UTC](https://discuss.elastic.co/t/mass-delete-by-query/16727/6 "2014-04-01T06:18:05Z")
In your use case, could the retention policy change for 89% document? If not, I would create one index for documents which could have a moving retention policy and use \_ttl. For monthly docs, I would use an index per month. If it's not the case, I think you should deal with \_ttl with a cost of higher merges. My 2 cents. -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 1 avr. 2014 à 08:03, slushi [kireetreddy@gmail.com](mailto:kireetreddy@gmail.com) a écrit : yes, unfortunately it’s not completely known at index time. I would need to keep the separate indices in sync when a retention policy change occurs. attempting this seems like it could open up a whole can of worms. > On Tuesday, April 1, 2014 1:58:04 AM UTC-4, David Pilato wrote: > If you know in advance which doc should be removed (i mean at index time), you should send the document to an index which should be entirely removed after a given period. > > Makes sense? > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 1 avr. 2014 à 00:00, slushi [kiree...@gmail.com](mailto:kiree...@gmail.com) a écrit : > > I attended an Elasticsearch meet up and at some point it was mentioned that TTL use is discouraged, but yes this would make a lot of sense here. Also the 1 year thing is really a guesstimate, we want to keep as much of that data as possible. I guess maybe with TTL you may not have as much control when the document deletion and possible segment merging? I am not that familiar with Elasticsearch performance stuff yet (we just started looking into using ES). > > > On Monday, March 31, 2014 5:52:28 PM UTC-4, Kevin Wang wrote: > > Why not use TTL for document? [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html) > > > > > On Tuesday, April 1, 2014 8:50:14 AM UTC+11, slushi wrote: > > > I have varying data retention requirements I am trying to balance (I am continuously indexing new documents): > > > 1% of my documents need to be kept forever > > > 10% need to be kept 1 year > > > the remainder needs to be kept for 1 month > > > I can easily set properties indicating the retention policy for each document and then periodically do a "delete by query". However, since the delete would remove 89% of the indexed documents, would there be any potential performance problems with this straightforward approach? I guess this is a YMMV type thing, but I was just wondering what the typical approach is here. Would it be necessary to perhaps filter the query to not affect so many documents at once? Would query performance be greatly impacted? > > > > > > The alternate approach I was thinking would be to create separate indices for each retention type. Cleanup would be easier, but unfortunately a document's retention policy can be upgraded/downgraded so that could be a little messy to keep consistent. > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9b685cff-e956-473a-935e-9546b2ea59b3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9b685cff-e956-473a-935e-9546b2ea59b3%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/eec089d7-0cef-4a9b-b53f-7dce55ad2bfd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/eec089d7-0cef-4a9b-b53f-7dce55ad2bfd%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/80DF8D6F-E0E8-46F3-BA7D-0D76D1B11E45%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/80DF8D6F-E0E8-46F3-BA7D-0D76D1B11E45%40pilato.fr). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:39am UTC](https://discuss.elastic.co/t/mass-delete-by-query/16727/7 "2017-07-06T01:39:07Z")