# TTL for documents **URL:** \ **Category:** Elasticsearch\ **Created:** [July 26, 2011, 9:48pm UTC](https://discuss.elastic.co/t/ttl-for-documents/4967 "2011-07-26T21:48:27Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Benjamin\_Deveze](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benjamin_deveze/32/1577_2.png) [@Benjamin\_Deveze](https://discuss.elastic.co/u/Benjamin_Deveze)\ **Post date:** [July 26, 2011, 9:48pm UTC](https://discuss.elastic.co/t/ttl-for-documents/4967/1 "2011-07-26T21:48:27Z")
A lot of documents naturally come with an expiration date. I think it would be nice to have a built-in support for a TTL/doc (with eventually default TTLs configurable per types/indices). I know disks are not expensive these days but it still is a common usage to use TTL for documents and it can be a very useful feature especially for people using ES as a key value storage. It is a pain to let the user trigger regularly some delete by query jobs to purge the data and I think it is a common enough use case to include it in the core of ES. Concerning the implementation of this feature I propose to introduce a special \_ttl field. When searching documents ES could hide expired ones (maybe adding a not query for expired docs in each query or something smarter). The documents could be really deleted and disk space liberated during segments merges. What do you think? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 27, 2011, 4:14am UTC](https://discuss.elastic.co/t/ttl-for-documents/4967/2 "2011-07-27T04:14:46Z")
Heya, Yes, its possible to add this feature. I think there is already an issue open for something similar... . Would love to hear what other people think... -shay.banon On Wed, Jul 27, 2011 at 12:48 AM, Benjamin Devèze \<[benjamin.deveze@gmail.com](mailto:benjamin.deveze@gmail.com) > wrote: > A lot of documents naturally come with an expiration date. I think it > would be nice to have a built-in support for a TTL/doc (with > eventually default TTLs configurable per types/indices). I know disks > are not expensive these days but it still is a common usage to use TTL > for documents and it can be a very useful feature especially for > people using ES as a key value storage. It is a pain to let the user > trigger regularly some delete by query jobs to purge the data and I > think it is a common enough use case to include it in the core of ES. > > Concerning the implementation of this feature I propose to introduce a > special \_ttl field. When searching documents ES could hide expired > ones (maybe adding a not query for expired docs in each query or > something smarter). The documents could be really deleted and disk > space liberated during segments merges. > > What do you think? ---
**Author:** ![Michel\_Conrad](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)\ **Post date:** [July 27, 2011, 8:22am UTC](https://discuss.elastic.co/t/ttl-for-documents/4967/3 "2011-07-27T08:22:44Z")
I've already been wondering why I couldn't send no more message to the list 😉 Well anyway here is what I wanted to post: +1 I would love to see this feature implemented as I am doing something similar on the client side and it would be much simpler if the server could take care of the expiration date. Best, Michel On Wed, Jul 27, 2011 at 6:14 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > Heya, > Yes, its possible to add this feature. I think there is already an issue > open for something similar... . Would love to hear what other people > think... > -shay.banon > > On Wed, Jul 27, 2011 at 12:48 AM, Benjamin Devèze > [benjamin.deveze@gmail.com](mailto:benjamin.deveze@gmail.com) wrote: > > > A lot of documents naturally come with an expiration date. I think it > > would be nice to have a built-in support for a TTL/doc (with > > eventually default TTLs configurable per types/indices). I know disks > > are not expensive these days but it still is a common usage to use TTL > > for documents and it can be a very useful feature especially for > > people using ES as a key value storage. It is a pain to let the user > > trigger regularly some delete by query jobs to purge the data and I > > think it is a common enough use case to include it in the core of ES. > > > > Concerning the implementation of this feature I propose to introduce a > > special \_ttl field. When searching documents ES could hide expired > > ones (maybe adding a not query for expired docs in each query or > > something smarter). The documents could be really deleted and disk > > space liberated during segments merges. > > > > What do you think? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 27, 2011, 8:41am UTC](https://discuss.elastic.co/t/ttl-for-documents/4967/4 "2011-07-27T08:41:00Z")
Yea, annoyed by it :). Btw, wanted to post another important note on managing expiring data. Another way of doing it, assuming it applies to the usecase, it to create an index per timespan, and then expire data by simply deleting old indices. The benefit of this usage pattern is the fact that deleting an index is much faster and has less strain on the system then deleting specific documents from an index (which will have to be merged out). For example, you could index log data into a single index, and have a ttl for it of 2 weeks. A better solution would be to create an index per week, and delete old indices that pass the 2 weeks mark. -shay.banon On Wed, Jul 27, 2011 at 11:22 AM, Michel Conrad \< [michel.conrad@trendiction.com](mailto:michel.conrad@trendiction.com)\> wrote: > I've already been wondering why I couldn't send no more message to the list > 😉 > Well anyway here is what I wanted to post: > > +1 > I would love to see this feature implemented as I am doing something > similar on the client side and it would be much simpler if the server > could take care of the expiration date. > > Best, > Michel > > On Wed, Jul 27, 2011 at 6:14 AM, Shay Banon > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > Heya, > > Yes, its possible to add this feature. I think there is already an > > issue > > open for something similar... . Would love to hear what other people > > think... > > -shay.banon > > > > On Wed, Jul 27, 2011 at 12:48 AM, Benjamin Devèze > > [benjamin.deveze@gmail.com](mailto:benjamin.deveze@gmail.com) wrote: > > > > > A lot of documents naturally come with an expiration date. I think it > > > would be nice to have a built-in support for a TTL/doc (with > > > eventually default TTLs configurable per types/indices). I know disks > > > are not expensive these days but it still is a common usage to use TTL > > > for documents and it can be a very useful feature especially for > > > people using ES as a key value storage. It is a pain to let the user > > > trigger regularly some delete by query jobs to purge the data and I > > > think it is a common enough use case to include it in the core of ES. > > > > > > Concerning the implementation of this feature I propose to introduce a > > > special \_ttl field. When searching documents ES could hide expired > > > ones (maybe adding a not query for expired docs in each query or > > > something smarter). The documents could be really deleted and disk > > > space liberated during segments merges. > > > > > > What do you think? ---
**Author:** ![Michel\_Conrad](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)\ **Post date:** [July 27, 2011, 8:55am UTC](https://discuss.elastic.co/t/ttl-for-documents/4967/5 "2011-07-27T08:55:37Z")
Thats exactly how I'm doing it at the moment. Although I think it would in some cases be convenient to have the possibility to specify a ttl while indexing, for instance if your expiring data is unregulary and sparsely on the time range. In this case it would be difficult to specify the timerange of the different indices, and if you want to, say keep docs of a maximum age of 6 month, I think it would be nice to specify the ttl for the docs while indexing, instead of periodically iterating over the results cleaning up manually. On Wed, Jul 27, 2011 at 10:41 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > Yea, annoyed by it :). Btw, wanted to post another important note on > managing expiring data. Another way of doing it, assuming it applies to the > usecase, it to create an index per timespan, and then expire data by simply > deleting old indices. The benefit of this usage pattern is the fact that > deleting an index is much faster and has less strain on the system then > deleting specific documents from an index (which will have to be merged > out). > For example, you could index log data into a single index, and have a ttl > for it of 2 weeks. A better solution would be to create an index per week, > and delete old indices that pass the 2 weeks mark. > -shay.banon > > On Wed, Jul 27, 2011 at 11:22 AM, Michel Conrad > [michel.conrad@trendiction.com](mailto:michel.conrad@trendiction.com) wrote: > > > I've already been wondering why I couldn't send no more message to the > > list 😉 > > Well anyway here is what I wanted to post: > > > > +1 > > I would love to see this feature implemented as I am doing something > > similar on the client side and it would be much simpler if the server > > could take care of the expiration date. > > > > Best, > > Michel > > > > On Wed, Jul 27, 2011 at 6:14 AM, Shay Banon > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > Heya, > > > Yes, its possible to add this feature. I think there is already an > > > issue > > > open for something similar... . Would love to hear what other people > > > think... > > > -shay.banon > > > > > > On Wed, Jul 27, 2011 at 12:48 AM, Benjamin Devèze > > > [benjamin.deveze@gmail.com](mailto:benjamin.deveze@gmail.com) wrote: > > > > > > > A lot of documents naturally come with an expiration date. I think it > > > > would be nice to have a built-in support for a TTL/doc (with > > > > eventually default TTLs configurable per types/indices). I know disks > > > > are not expensive these days but it still is a common usage to use TTL > > > > for documents and it can be a very useful feature especially for > > > > people using ES as a key value storage. It is a pain to let the user > > > > trigger regularly some delete by query jobs to purge the data and I > > > > think it is a common enough use case to include it in the core of ES. > > > > > > > > Concerning the implementation of this feature I propose to introduce a > > > > special \_ttl field. When searching documents ES could hide expired > > > > ones (maybe adding a not query for expired docs in each query or > > > > something smarter). The documents could be really deleted and disk > > > > space liberated during segments merges. > > > > > > > > What do you think? ---
**Author:** ![Benjamin\_Deveze](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benjamin_deveze/32/1577_2.png) [@Benjamin\_Deveze](https://discuss.elastic.co/u/Benjamin_Deveze)\ **Post date:** [July 27, 2011, 9:00am UTC](https://discuss.elastic.co/t/ttl-for-documents/4967/6 "2011-07-27T09:00:11Z")
Yeah I agree on the index per timespan approach, it is an efficient way to handle expiring data for logs and things like that but it doesn't really fit my use cases: - the user still has to manage expired indices deletion by external jobs which is of course easy but not really nice - it is not really flexible because you have to choose your time range a priori. In my use case I would like to be able to dynamically change the TTL of an indexed doc and I don't want to have to delete it from an index and reindex it to another one fitting the new TTL - that can lead to a lot of indices (think for example one index/user + divide each index by time range...) which add an overhead ---
**Author:** ![Benjamin\_Deveze](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benjamin_deveze/32/1577_2.png) [@Benjamin\_Deveze](https://discuss.elastic.co/u/Benjamin_Deveze)\ **Post date:** [July 27, 2011, 12:27pm UTC](https://discuss.elastic.co/t/ttl-for-documents/4967/7 "2011-07-27T12:27:21Z")
If there are other people interested and if we can agree here to a good way to implement it I am quite willing to implement it. Kimchy do you have special recommendations, concerns about the implementation? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 27, 2011, 5:28pm UTC](https://discuss.elastic.co/t/ttl-for-documents/4967/8 "2011-07-27T17:28:13Z")
I think we identified two different implementations: 1. The first, is one that I have been thinking for a long time, and its automatic rolling of indices. Basically, utilizing the index templates notion, one can define an index rolling strategy (time based can be the first one). When indexing, we can check if a rollover is needed, and if so, we can create a new index and index the data into it. The fact that its built on top of index templates will automatically support custom settings and mappings for the indices created. 2. TTL per document in the index. That one is a bit more tricky as it requires to think where the TTL will be stored. It can be stored in the document, but then it requires reindexing whenever it changes. It will also require a process that periodically evicts old documents. On Wed, Jul 27, 2011 at 3:27 PM, Benjamin Devèze [benjamin.deveze@gmail.com](mailto:benjamin.deveze@gmail.com)wrote: > If there are other people interested and if we can agree here to a good way > to implement it I am quite willing to implement it. Kimchy do you have > special recommendations, concerns about the implementation? ---
**Author:** ![Mahendra\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mahendra_m/32/3128_2.png) [@Mahendra\_M](https://discuss.elastic.co/u/Mahendra_M)\ **Post date:** [July 28, 2011, 6:12am UTC](https://discuss.elastic.co/t/ttl-for-documents/4967/9 "2011-07-28T06:12:03Z")
(Sending to new mailing list) +1 to this feature. It will help in my scenario. I have docs in CouchDB which have publication and expiry timestamps in them. I expose Elasticsearch as a query layer to users for these docs. I have a celery (python) job which keeps syncing (POST/DELETE) these docs to Elasticsearch at appropriate times. Typically there is a 2 - 5 minute delay in these operations ( pubish\_time + x\_minutes ). It is OK for me to publish the doc to Elasticsearch with a delay, but withdrawing with a delay is a bit painful. If a \_ttl field is supported, it will make withdrawing docs easier and almost realtime. Regards, Mahendra On Wed, Jul 27, 2011 at 9:44 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Heya, > > Yes, its possible to add this feature. I think there is already an issue > open for something similar... . Would love to hear what other people > think... > > -shay.banon > > On Wed, Jul 27, 2011 at 12:48 AM, Benjamin Devèze \< > [benjamin.deveze@gmail.com](mailto:benjamin.deveze@gmail.com)\> wrote: > > > A lot of documents naturally come with an expiration date. I think it > > would be nice to have a built-in support for a TTL/doc (with > > eventually default TTLs configurable per types/indices). I know disks > > are not expensive these days but it still is a common usage to use TTL > > for documents and it can be a very useful feature especially for > > people using ES as a key value storage. It is a pain to let the user > > trigger regularly some delete by query jobs to purge the data and I > > think it is a common enough use case to include it in the core of ES. > > > > Concerning the implementation of this feature I propose to introduce a > > special \_ttl field. When searching documents ES could hide expired > > ones (maybe adding a not query for expired docs in each query or > > something smarter). The documents could be really deleted and disk > > space liberated during segments merges. > > > > What do you think? -- Mahendra [http://twitter.com/mahendra](http://twitter.com/mahendra) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:58am UTC](https://discuss.elastic.co/t/ttl-for-documents/4967/10 "2017-07-06T03:58:59Z")