# TTL issue of ES 0.18.7 **URL:** \ **Category:** Elasticsearch\ **Created:** [June 4, 2012, 9:20am UTC](https://discuss.elastic.co/t/ttl-issue-of-es-0-18-7/7956 "2012-06-04T09:20:26Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![yangzhaoxia](https://avatars.discourse-cdn.com/v4/letter/y/7cd45c/32.png) [@yangzhaoxia](https://discuss.elastic.co/u/yangzhaoxia)\ **Post date:** [June 4, 2012, 9:20am UTC](https://discuss.elastic.co/t/ttl-issue-of-es-0-18-7/7956/1 "2012-06-04T09:20:26Z")
My case is using logstash/rabbitmq/graylog2. I have to use ES 0.18.7 to store messages for Graylog2 since ES 0.18.7 is required by logstash. The index created by Graylog2 server is "graylog2". All the messages are indexed by "graylog2". Now I want to purge messages with 3 minutes expiration (3m is for test), so I want to use TTL feature of index. {code} curl -XPOST 'localhost:9200/graylog2/\_close' {"ok":true,"acknowledged":true} curl -XPUT [http://localhost:9200/graylog2/\_settings](http://localhost:9200/graylog2/_settings) -d '{"graylog2": {"\_ttl" : { "enabled" : true, "default" : "3m"},"settings": {"index.number\_of\_shards":"5","index.number\_of\_replicas":"1"}}}' curl -XGET [http://localhost:9200/graylog2/\_settings](http://localhost:9200/graylog2/_settings) {"graylog2":{"settings": {"index.number\_of\_shards":"5","index.number\_of\_replicas":"1","index.graylog2.\_ttl.enabled":"true","index.graylog2.settings.index.number\_of\_shards":"5","index.graylog2.\_ttl.default":"3d","index.graylog2.settings.index.number\_of\_replicas":"1"}}} # curl -XPOST 'localhost:9200/graylog2/\_open' {code} I didn't see any messages deleted after 40 minutes' observation(messages kept growing and never deleted from Graylog2 Web GUI) but I thought new messages would be deleted once ttl over 3m. Could you help me? Besides, I set TTL for index level, does this mean the whole index would be deleted after the ttl period, or the messages of the index would be purged? I also tried to set ttl according to [http://dev.nuclearrooster.com/2012/05/15/enable-elasticsearch-ttl-for-graylog2-messages/](http://dev.nuclearrooster.com/2012/05/15/enable-elasticsearch-ttl-for-graylog2-messages/) but nothing changed. -- check the mapping of graylog2 # curl [http://localhost:9200/graylog2/message/\_mapping](http://localhost:9200/graylog2/message/_mapping) {"message":{"dynamic\_templates":[{"store\_generic":{"mapping": {"index":"not\_analyzed"},"match":"\*"}}],"properties":{"message": {"analyzer":"whitespace","type":"string"},"level": {"type":"long"},"host": {"index":"not\_analyzed","type":"string"},"facility": {"index":"not\_analyzed","type":"string"},"file": {"index":"not\_analyzed","type":"string"},"full\_message": {"analyzer":"whitespace","type":"string"},"line": {"type":"long"},"created\_at":{"type":"double"},"streams": {"index":"not\_analyzed","type":"string"}}}}[ -- add ttl setting # curl [http://localhost:9200/graylog2/message/\_mapping](http://localhost:9200/graylog2/message/_mapping) -XPUT -d '{"message":{"dynamic\_templates":[{"store\_generic":{"mapping": {"index":"not\_analyzed"},"match":"\*"}}],"\_ttl" : { "enabled" : true, "default" : "30d"},"properties":{"message": {"analyzer":"whitespace","type":"string"},"level": {"type":"long"},"host": {"index":"not\_analyzed","type":"string"},"facility": {"index":"not\_analyzed","type":"string"},"file": {"index":"not\_analyzed","type":"string"},"full\_message": {"analyzer":"whitespace","type":"string"},"line": {"type":"long"},"created\_at":{"type":"double"},"streams": {"index":"not\_analyzed","type":"string"}}}}' {"ok":true,"acknowledged":true} -- check mapping setting after adding ttl, but nothing changed # curl [http://localhost:9200/graylog2/message/\_mapping](http://localhost:9200/graylog2/message/_mapping) {"message":{"dynamic\_templates":[{"store\_generic":{"mapping": {"index":"not\_analyzed"},"match":"\*"}}],"properties":{"message": {"analyzer":"whitespace","type":"string"},"level": {"type":"long"},"host": {"index":"not\_analyzed","type":"string"},"facility": {"index":"not\_analyzed","type":"string"},"file": {"index":"not\_analyzed","type":"string"},"full\_message": {"analyzer":"whitespace","type":"string"},"line": {"type":"long"},"created\_at":{"type":"double"},"streams": {"index":"not\_analyzed","type":"string"}}}} Regards, ---
**Author:** ![Benjamin\_Deveze](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benjamin_deveze/32/1577_2.png) [@Benjamin\_Deveze](https://discuss.elastic.co/u/Benjamin_Deveze)\ **Post date:** [June 4, 2012, 9:36am UTC](https://discuss.elastic.co/t/ttl-issue-of-es-0-18-7/7956/2 "2012-06-04T09:36:09Z")
Heya, 1. if I remember well the dynamic mapping update of an index default TTL was added after ES 0.18.7 that is certainly why you don't see changes and why new indexed messages are not properly purged after 3 minutes 2. Only the messages of the index will be purged not the whole index Hope this help On Mon, Jun 4, 2012 at 11:20 AM, yang zhaoxia [yangzhaoxia@gmail.com](mailto:yangzhaoxia@gmail.com) wrote: > My case is using logstash/rabbitmq/graylog2. > I have to use ES 0.18.7 to store messages for Graylog2 since ES 0.18.7 > is required by logstash. > The index created by Graylog2 server is "graylog2". All the messages > are indexed by "graylog2". > > Now I want to purge messages with 3 minutes expiration (3m is for > test), so I want to use TTL feature of index. > {code} > curl -XPOST 'localhost:9200/graylog2/\_close' > {"ok":true,"acknowledged":true} > > curl -XPUT [http://localhost:9200/graylog2/\_settings](http://localhost:9200/graylog2/_settings) -d '{"graylog2": > {"\_ttl" : { "enabled" : true, "default" : "3m"},"settings": > {"index.number\_of\_shards":"5","index.number\_of\_replicas":"1"}}}' > > curl -XGET [http://localhost:9200/graylog2/\_settings](http://localhost:9200/graylog2/_settings) > {"graylog2":{"settings": > {"index.number\_of\_shards":"5","index.number\_of\_replicas":"1","index.graylog2.\_ttl.enabled":"true","index.graylog2.settings.index.number\_of\_shards":"5","index.graylog2.\_ttl.default":"3d","index.graylog2.settings.index.number\_of\_replicas":"1"}}} > > # curl -XPOST 'localhost:9200/graylog2/\_open' > > {code} > > I didn't see any messages deleted after 40 minutes' > observation(messages kept growing and never deleted from Graylog2 Web > GUI) but I thought new messages would be deleted once ttl over 3m. > > Could you help me? > Besides, I set TTL for index level, does this mean the whole index > would be deleted after the ttl period, or the messages of the index > would be purged? > > I also tried to set ttl according to > [http://dev.nuclearrooster.com/2012/05/15/enable-elasticsearch-ttl-for-graylog2-messages/](http://dev.nuclearrooster.com/2012/05/15/enable-elasticsearch-ttl-for-graylog2-messages/) > but nothing changed. > > -- check the mapping of graylog2 > > # curl [http://localhost:9200/graylog2/message/\_mapping](http://localhost:9200/graylog2/message/_mapping) > > {"message":{"dynamic\_templates":[{"store\_generic":{"mapping": > {"index":"not\_analyzed"},"match":"\*"}}],"properties":{"message": > {"analyzer":"whitespace","type":"string"},"level": > {"type":"long"},"host": > {"index":"not\_analyzed","type":"string"},"facility": > {"index":"not\_analyzed","type":"string"},"file": > {"index":"not\_analyzed","type":"string"},"full\_message": > {"analyzer":"whitespace","type":"string"},"line": > {"type":"long"},"created\_at":{"type":"double"},"streams": > {"index":"not\_analyzed","type":"string"}}}}[ > > -- add ttl setting > > # curl [http://localhost:9200/graylog2/message/\_mapping](http://localhost:9200/graylog2/message/_mapping) -XPUT -d > > '{"message":{"dynamic\_templates":[{"store\_generic":{"mapping": > {"index":"not\_analyzed"},"match":"\*"}}],"\_ttl" : { "enabled" : true, > "default" : "30d"},"properties":{"message": > {"analyzer":"whitespace","type":"string"},"level": > {"type":"long"},"host": > {"index":"not\_analyzed","type":"string"},"facility": > {"index":"not\_analyzed","type":"string"},"file": > {"index":"not\_analyzed","type":"string"},"full\_message": > {"analyzer":"whitespace","type":"string"},"line": > {"type":"long"},"created\_at":{"type":"double"},"streams": > {"index":"not\_analyzed","type":"string"}}}}' > {"ok":true,"acknowledged":true} > > -- check mapping setting after adding ttl, but nothing changed > > # curl [http://localhost:9200/graylog2/message/\_mapping](http://localhost:9200/graylog2/message/_mapping) > > {"message":{"dynamic\_templates":[{"store\_generic":{"mapping": > {"index":"not\_analyzed"},"match":"\*"}}],"properties":{"message": > {"analyzer":"whitespace","type":"string"},"level": > {"type":"long"},"host": > {"index":"not\_analyzed","type":"string"},"facility": > {"index":"not\_analyzed","type":"string"},"file": > {"index":"not\_analyzed","type":"string"},"full\_message": > {"analyzer":"whitespace","type":"string"},"line": > {"type":"long"},"created\_at":{"type":"double"},"streams": > {"index":"not\_analyzed","type":"string"}}}} > > Regards, -- Benjamin DEVEZE ---
**Author:** ![yangzhaoxia](https://avatars.discourse-cdn.com/v4/letter/y/7cd45c/32.png) [@yangzhaoxia](https://discuss.elastic.co/u/yangzhaoxia)\ **Post date:** [June 4, 2012, 9:49am UTC](https://discuss.elastic.co/t/ttl-issue-of-es-0-18-7/7956/3 "2012-06-04T09:49:43Z")
Thanks a lot, Benjamin. At first I didn't run index close/open, then setting XPUT would cause below warning in the ES server side: [graylog2] ignoring non dynamic index level settings for open indices: [index.graylog2.\_ttl.default, index.graylog2.\_ttl.enabled, index.graylog2.settings.index.number\_of\_replicas, index.graylog2.settings.index.number\_of\_shards] From this warning, I thought it should support TTL changes after running index close/open. That's a pity. On Jun 4, 5:36 pm, Benjamin Devèze [benjamin.dev...@gmail.com](mailto:benjamin.dev...@gmail.com) wrote: > Heya, > > 1. if I remember well the dynamic mapping update of an index default > TTL was added after ES 0.18.7 that is certainly why you don't see > changes and why new indexed messages are not properly purged after 3 > minutes > 2. Only the messages of the index will be purged not the whole index > > Hope this help > > On Mon, Jun 4, 2012 at 11:20 AM, yang zhaoxia [yangzhao...@gmail.com](mailto:yangzhao...@gmail.com) wrote: > > > My case is using logstash/rabbitmq/graylog2. > > I have to use ES 0.18.7 to store messages for Graylog2 since ES 0.18.7 > > is required by logstash. > > The index created by Graylog2 server is "graylog2". All the messages > > are indexed by "graylog2". > > > Now I want to purge messages with 3 minutes expiration (3m is for > > test), so I want to use TTL feature of index. > > {code} > > curl -XPOST 'localhost:9200/graylog2/\_close' > > {"ok":true,"acknowledged":true} > > > curl -XPUThttp://localhost:9200/graylog2/\_settings-d '{"graylog2": > > {"\_ttl" : { "enabled" : true, "default" : "3m"},"settings": > > {"index.number\_of\_shards":"5","index.number\_of\_replicas":"1"}}}' > > > curl -XGEThttp://localhost:9200/graylog2/\_settings > > {"graylog2":{"settings": > > {"index.number\_of\_shards":"5","index.number\_of\_replicas":"1","index.graylog 2.\_ttl.enabled":"true","index.graylog2.settings.index.number\_of\_shards":"5" ,"index.graylog2.\_ttl.default":"3d","index.graylog2.settings.index.number\_o f\_replicas":"1"}}} > > > # curl -XPOST 'localhost:9200/graylog2/\_open' > > > > {code} > > > I didn't see any messages deleted after 40 minutes' > > observation(messages kept growing and never deleted from Graylog2 Web > > GUI) but I thought new messages would be deleted once ttl over 3m. > > > Could you help me? > > Besides, I set TTL for index level, does this mean the whole index > > would be deleted after the ttl period, or the messages of the index > > would be purged? > > > I also tried to set ttl according to > > [http://dev.nuclearrooster.com/2012/05/15/enable-elasticsearch-ttl-for](http://dev.nuclearrooster.com/2012/05/15/enable-elasticsearch-ttl-for)... > > but nothing changed. > > > -- check the mapping of graylog2 > > > > # curlhttp://localhost:9200/graylog2/message/\_mapping > > > > {"message":{"dynamic\_templates":[{"store\_generic":{"mapping": > > {"index":"not\_analyzed"},"match":"\*"}}],"properties":{"message": > > {"analyzer":"whitespace","type":"string"},"level": > > {"type":"long"},"host": > > {"index":"not\_analyzed","type":"string"},"facility": > > {"index":"not\_analyzed","type":"string"},"file": > > {"index":"not\_analyzed","type":"string"},"full\_message": > > {"analyzer":"whitespace","type":"string"},"line": > > {"type":"long"},"created\_at":{"type":"double"},"streams": > > {"index":"not\_analyzed","type":"string"}}}}[ > > > -- add ttl setting > > > > # curlhttp://localhost:9200/graylog2/message/\_mapping-XPUT -d > > > > '{"message":{"dynamic\_templates":[{"store\_generic":{"mapping": > > {"index":"not\_analyzed"},"match":"\*"}}],"\_ttl" : { "enabled" : true, > > "default" : "30d"},"properties":{"message": > > {"analyzer":"whitespace","type":"string"},"level": > > {"type":"long"},"host": > > {"index":"not\_analyzed","type":"string"},"facility": > > {"index":"not\_analyzed","type":"string"},"file": > > {"index":"not\_analyzed","type":"string"},"full\_message": > > {"analyzer":"whitespace","type":"string"},"line": > > {"type":"long"},"created\_at":{"type":"double"},"streams": > > {"index":"not\_analyzed","type":"string"}}}}' > > {"ok":true,"acknowledged":true} > > > -- check mapping setting after adding ttl, but nothing changed > > > > # curlhttp://localhost:9200/graylog2/message/\_mapping > > > > {"message":{"dynamic\_templates":[{"store\_generic":{"mapping": > > {"index":"not\_analyzed"},"match":"\*"}}],"properties":{"message": > > {"analyzer":"whitespace","type":"string"},"level": > > {"type":"long"},"host": > > {"index":"not\_analyzed","type":"string"},"facility": > > {"index":"not\_analyzed","type":"string"},"file": > > {"index":"not\_analyzed","type":"string"},"full\_message": > > {"analyzer":"whitespace","type":"string"},"line": > > {"type":"long"},"created\_at":{"type":"double"},"streams": > > {"index":"not\_analyzed","type":"string"}}}} > > > Regards, > > -- > Benjamin DEVEZE ---
**Author:** ![yangzhaoxia](https://avatars.discourse-cdn.com/v4/letter/y/7cd45c/32.png) [@yangzhaoxia](https://discuss.elastic.co/u/yangzhaoxia)\ **Post date:** [June 7, 2012, 9:14am UTC](https://discuss.elastic.co/t/ttl-issue-of-es-0-18-7/7956/4 "2012-06-07T09:14:46Z")
Since ES 0.18.7 doesn't supppor this TTL dynamic changes, I finally made it work as expected via Graylog2 Message Retention functionality. Thanks. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:25am UTC](https://discuss.elastic.co/t/ttl-issue-of-es-0-18-7/7956/5 "2017-07-06T03:25:24Z")