# Expired documents not purged **URL:** \ **Category:** Elasticsearch\ **Created:** [May 6, 2013, 7:25am UTC](https://discuss.elastic.co/t/expired-documents-not-purged/11828 "2013-05-06T07:25:59Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Stephane\_Cottin](https://avatars.discourse-cdn.com/v4/letter/s/9de0a6/32.png) [@Stephane\_Cottin](https://discuss.elastic.co/u/Stephane_Cottin)\ **Post date:** [May 6, 2013, 7:25am UTC](https://discuss.elastic.co/t/expired-documents-not-purged/11828/1 "2013-05-06T07:25:59Z")
Hi, I'm using ES 0.90 with graylog2, expired documents in recent index are not purged. appear ES related to me, is this a bug or a misconfiguration ? how can i manually purge old entries ? thx \_settings : { "graylog2\_recent" : { "settings" : { "index.store.type" : "niofs", "index.number\_of\_replicas" : "0", "index.number\_of\_shards" : "4", "index.version.created" : "900099" } } } \_mapping : { "graylog2\_recent" : { "message" : { "dynamic\_templates" : [ { "store\_generic" : { "mapping" : { "index" : "not\_analyzed" }, "match" : "\*" } } ], "\_ttl" : { "enabled" : true }, "\_source" : { "compress" : true }, [...] [...]/graylog2\_recent/\_search?pretty=true&fields=\_ttl,\_timestamp { "took" : 8, "timed\_out" : false, "\_shards" : { "total" : 4, "successful" : 4, "failed" : 0 }, "hits" : { "total" : 463837, "max\_score" : 1.0, "hits" : [ { "\_index" : "graylog2\_recent", "\_type" : "message", "\_id" : "bfbebd30-b5db-11e2-aecd-00163e783829", "\_score" : 1.0, "fields" : { "\_ttl" : -23976963, "\_timestamp" : "06/May/2013:01:30:19 +0200" } }, { "\_index" : "graylog2\_recent", "\_type" : "message", "\_id" : "bfcbb580-b5db-11e2-aecd-00163e783829", "\_score" : 1.0, "fields" : { "\_ttl" : -23976878, "\_timestamp" : "06/May/2013:01:30:19 +0200" } }, { "\_index" : "graylog2\_recent", "\_type" : "message", "\_id" : "bfd37db0-b5db-11e2-aecd-00163e783829", "\_score" : 1.0, "fields" : { "\_ttl" : -23976828 } }, { "\_index" : "graylog2\_recent", "\_type" : "message", "\_id" : "bfeaaf30-b5db-11e2-aecd-00163e783829", "\_score" : 1.0, "fields" : { "\_ttl" : -23976675, "\_timestamp" : "06/May/2013:01:30:19 +0200" } }, { [...] -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Stephane\_Cottin](https://avatars.discourse-cdn.com/v4/letter/s/9de0a6/32.png) [@Stephane\_Cottin](https://discuss.elastic.co/u/Stephane_Cottin)\ **Post date:** [May 8, 2013, 8:04am UTC](https://discuss.elastic.co/t/expired-documents-not-purged/11828/2 "2013-05-08T08:04:15Z")
upgrade from openjdk 6 to openjdk 7 fix it. Le lundi 6 mai 2013 09:25:59 UTC+2, Stéphane Cottin a écrit : > Hi, > > I'm using ES 0.90 with graylog2, expired documents in recent index are not > purged. > appear ES related to me, is this a bug or a misconfiguration ? > how can i manually purge old entries ? > > thx > > \_settings : > > { > "graylog2\_recent" : { > "settings" : { > "index.store.type" : "niofs", > "index.number\_of\_replicas" : "0", > "index.number\_of\_shards" : "4", > "index.version.created" : "900099" > } > } > } > > \_mapping : > > { > "graylog2\_recent" : { > "message" : { > "dynamic\_templates" : [ { > "store\_generic" : { > "mapping" : { > "index" : "not\_analyzed" > }, > "match" : "\*" > } > } ], > "\_ttl" : { > "enabled" : true > }, > "\_source" : { > "compress" : true > }, > [...] > > [...]/graylog2\_recent/\_search?pretty=true&fields=\_ttl,\_timestamp > { > "took" : 8, > "timed\_out" : false, > "\_shards" : { > "total" : 4, > "successful" : 4, > "failed" : 0 > }, > "hits" : { > "total" : 463837, > "max\_score" : 1.0, > "hits" : [ { > "\_index" : "graylog2\_recent", > "\_type" : "message", > "\_id" : "bfbebd30-b5db-11e2-aecd-00163e783829", > "\_score" : 1.0, > "fields" : { > "\_ttl" : -23976963, > "\_timestamp" : "06/May/2013:01:30:19 +0200" > } > }, { > "\_index" : "graylog2\_recent", > "\_type" : "message", > "\_id" : "bfcbb580-b5db-11e2-aecd-00163e783829", > "\_score" : 1.0, > "fields" : { > "\_ttl" : -23976878, > "\_timestamp" : "06/May/2013:01:30:19 +0200" > } > }, { > "\_index" : "graylog2\_recent", > "\_type" : "message", > "\_id" : "bfd37db0-b5db-11e2-aecd-00163e783829", > "\_score" : 1.0, > "fields" : { > "\_ttl" : -23976828 > } > }, { > "\_index" : "graylog2\_recent", > "\_type" : "message", > "\_id" : "bfeaaf30-b5db-11e2-aecd-00163e783829", > "\_score" : 1.0, > "fields" : { > "\_ttl" : -23976675, > "\_timestamp" : "06/May/2013:01:30:19 +0200" > } > }, { > [...] -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Stephane\_Cottin](https://avatars.discourse-cdn.com/v4/letter/s/9de0a6/32.png) [@Stephane\_Cottin](https://discuss.elastic.co/u/Stephane_Cottin)\ **Post date:** [May 10, 2013, 8:06am UTC](https://discuss.elastic.co/t/expired-documents-not-purged/11828/3 "2013-05-10T08:06:52Z")
I was wrong, java update did not fix anything. ttl cleaning seems broken in 0.90. running with debug log level show purge loops are fired, but they don't delete anything. as a workaround, i'm running a cron job once per hour : #!/bin/sh curl -XDELETE [http://127.0.0.1:9200/graylog2\_recent/\_query](http://127.0.0.1:9200/graylog2_recent/_query) -d "{"range":{"\_ttl":{"to":$(date +%s)000}}}" curl [http://127.0.0.1:9200/graylog2\_recent/\_optimize](http://127.0.0.1:9200/graylog2_recent/_optimize) not elegant, but works Le mercredi 8 mai 2013 10:04:15 UTC+2, Stéphane Cottin a écrit : > upgrade from openjdk 6 to openjdk 7 fix it. > > Le lundi 6 mai 2013 09:25:59 UTC+2, Stéphane Cottin a écrit : > > > Hi, > > > > I'm using ES 0.90 with graylog2, expired documents in recent index are > > not purged. > > appear ES related to me, is this a bug or a misconfiguration ? > > how can i manually purge old entries ? > > > > thx > > > > \_settings : > > > > { > > "graylog2\_recent" : { > > "settings" : { > > "index.store.type" : "niofs", > > "index.number\_of\_replicas" : "0", > > "index.number\_of\_shards" : "4", > > "index.version.created" : "900099" > > } > > } > > } > > > > \_mapping : > > > > { > > "graylog2\_recent" : { > > "message" : { > > "dynamic\_templates" : [ { > > "store\_generic" : { > > "mapping" : { > > "index" : "not\_analyzed" > > }, > > "match" : "\*" > > } > > } ], > > "\_ttl" : { > > "enabled" : true > > }, > > "\_source" : { > > "compress" : true > > }, > > [...] > > > > [...]/graylog2\_recent/\_search?pretty=true&fields=\_ttl,\_timestamp > > { > > "took" : 8, > > "timed\_out" : false, > > "\_shards" : { > > "total" : 4, > > "successful" : 4, > > "failed" : 0 > > }, > > "hits" : { > > "total" : 463837, > > "max\_score" : 1.0, > > "hits" : [ { > > "\_index" : "graylog2\_recent", > > "\_type" : "message", > > "\_id" : "bfbebd30-b5db-11e2-aecd-00163e783829", > > "\_score" : 1.0, > > "fields" : { > > "\_ttl" : -23976963, > > "\_timestamp" : "06/May/2013:01:30:19 +0200" > > } > > }, { > > "\_index" : "graylog2\_recent", > > "\_type" : "message", > > "\_id" : "bfcbb580-b5db-11e2-aecd-00163e783829", > > "\_score" : 1.0, > > "fields" : { > > "\_ttl" : -23976878, > > "\_timestamp" : "06/May/2013:01:30:19 +0200" > > } > > }, { > > "\_index" : "graylog2\_recent", > > "\_type" : "message", > > "\_id" : "bfd37db0-b5db-11e2-aecd-00163e783829", > > "\_score" : 1.0, > > "fields" : { > > "\_ttl" : -23976828 > > } > > }, { > > "\_index" : "graylog2\_recent", > > "\_type" : "message", > > "\_id" : "bfeaaf30-b5db-11e2-aecd-00163e783829", > > "\_score" : 1.0, > > "fields" : { > > "\_ttl" : -23976675, > > "\_timestamp" : "06/May/2013:01:30:19 +0200" > > } > > }, { > > [...] -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 10, 2013, 8:21am UTC](https://discuss.elastic.co/t/expired-documents-not-purged/11828/4 "2013-05-10T08:21:00Z")
What does "\_ttl" : -23976963 mean? I think that the 'error' comes from here. My 2 cents. -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 10 mai 2013 à 10:06, Stéphane Cottin [vixnsgm@gmail.com](mailto:vixnsgm@gmail.com) a écrit : I was wrong, java update did not fix anything. ttl cleaning seems broken in 0.90. running with debug log level show purge loops are fired, but they don't delete anything. as a workaround, i'm running a cron job once per hour : #!/bin/sh curl -XDELETE [http://127.0.0.1:9200/graylog2\_recent/\_query](http://127.0.0.1:9200/graylog2_recent/_query) -d "{"range":{"\_ttl":{"to":$(date +%s)000}}}" curl [http://127.0.0.1:9200/graylog2\_recent/\_optimize](http://127.0.0.1:9200/graylog2_recent/_optimize) not elegant, but works Le mercredi 8 mai 2013 10:04:15 UTC+2, Stéphane Cottin a écrit : > upgrade from openjdk 6 to openjdk 7 fix it. > > Le lundi 6 mai 2013 09:25:59 UTC+2, Stéphane Cottin a écrit : > > > Hi, > > > > I'm using ES 0.90 with graylog2, expired documents in recent index are not purged. > > appear ES related to me, is this a bug or a misconfiguration ? > > how can i manually purge old entries ? > > > > thx > > > > \_settings : > > > > { > > "graylog2\_recent" : { > > "settings" : { > > "index.store.type" : "niofs", > > "index.number\_of\_replicas" : "0", > > "index.number\_of\_shards" : "4", > > "index.version.created" : "900099" > > } > > } > > } > > > > \_mapping : > > > > { > > "graylog2\_recent" : { > > "message" : { > > "dynamic\_templates" : [ { > > "store\_generic" : { > > "mapping" : { > > "index" : "not\_analyzed" > > }, > > "match" : "\*" > > } > > } ], > > "\_ttl" : { > > "enabled" : true > > }, > > "\_source" : { > > "compress" : true > > }, > > [...] > > > > [...]/graylog2\_recent/\_search?pretty=true&fields=\_ttl,\_timestamp > > { > > "took" : 8, > > "timed\_out" : false, > > "\_shards" : { > > "total" : 4, > > "successful" : 4, > > "failed" : 0 > > }, > > "hits" : { > > "total" : 463837, > > "max\_score" : 1.0, > > "hits" : [ { > > "\_index" : "graylog2\_recent", > > "\_type" : "message", > > "\_id" : "bfbebd30-b5db-11e2-aecd-00163e783829", > > "\_score" : 1.0, > > "fields" : { > > "\_ttl" : -23976963, > > "\_timestamp" : "06/May/2013:01:30:19 +0200" > > } > > }, { > > "\_index" : "graylog2\_recent", > > "\_type" : "message", > > "\_id" : "bfcbb580-b5db-11e2-aecd-00163e783829", > > "\_score" : 1.0, > > "fields" : { > > "\_ttl" : -23976878, > > "\_timestamp" : "06/May/2013:01:30:19 +0200" > > } > > }, { > > "\_index" : "graylog2\_recent", > > "\_type" : "message", > > "\_id" : "bfd37db0-b5db-11e2-aecd-00163e783829", > > "\_score" : 1.0, > > "fields" : { > > "\_ttl" : -23976828 > > } > > }, { > > "\_index" : "graylog2\_recent", > > "\_type" : "message", > > "\_id" : "bfeaaf30-b5db-11e2-aecd-00163e783829", > > "\_score" : 1.0, > > "fields" : { > > "\_ttl" : -23976675, > > "\_timestamp" : "06/May/2013:01:30:19 +0200" > > } > > }, { > > [...] > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Stephane\_Cottin](https://avatars.discourse-cdn.com/v4/letter/s/9de0a6/32.png) [@Stephane\_Cottin](https://discuss.elastic.co/u/Stephane_Cottin)\ **Post date:** [May 10, 2013, 8:29am UTC](https://discuss.elastic.co/t/expired-documents-not-purged/11828/5 "2013-05-10T08:29:52Z")
Le 10 mai 2013 à 10:21, David Pilato [david@pilato.fr](mailto:david@pilato.fr) a écrit : > What does "\_ttl" : -23976963 mean? > I think that the 'error' comes from here. It was my first though, but as the delete query works, so internal ttl values are correct -23976963 is just a long displayed as int afaik > My 2 cents. > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 10 mai 2013 à 10:06, Stéphane Cottin [vixnsgm@gmail.com](mailto:vixnsgm@gmail.com) a écrit : > > I was wrong, java update did not fix anything. > > ttl cleaning seems broken in 0.90. > running with debug log level show purge loops are fired, but they don't delete anything. > > as a workaround, i'm running a cron job once per hour : > > #!/bin/sh > curl -XDELETE [http://127.0.0.1:9200/graylog2\_recent/\_query](http://127.0.0.1:9200/graylog2_recent/_query) -d "{"range":{"\_ttl":{"to":$(date +%s)000}}}" > curl [http://127.0.0.1:9200/graylog2\_recent/\_optimize](http://127.0.0.1:9200/graylog2_recent/_optimize) > > not elegant, but works > > Le mercredi 8 mai 2013 10:04:15 UTC+2, Stéphane Cottin a écrit : > upgrade from openjdk 6 to openjdk 7 fix it. > > Le lundi 6 mai 2013 09:25:59 UTC+2, Stéphane Cottin a écrit : > Hi, > > I'm using ES 0.90 with graylog2, expired documents in recent index are not purged. > appear ES related to me, is this a bug or a misconfiguration ? > how can i manually purge old entries ? > > thx > > \_settings : > > { > "graylog2\_recent" : { > "settings" : { > "index.store.type" : "niofs", > "index.number\_of\_replicas" : "0", > "index.number\_of\_shards" : "4", > "index.version.created" : "900099" > } > } > } > > \_mapping : > > { > "graylog2\_recent" : { > "message" : { > "dynamic\_templates" : [ { > "store\_generic" : { > "mapping" : { > "index" : "not\_analyzed" > }, > "match" : "\*" > } > } ], > "\_ttl" : { > "enabled" : true > }, > "\_source" : { > "compress" : true > }, > [...] > > [...]/graylog2\_recent/\_search?pretty=true&fields=\_ttl,\_timestamp > { > "took" : 8, > "timed\_out" : false, > "\_shards" : { > "total" : 4, > "successful" : 4, > "failed" : 0 > }, > "hits" : { > "total" : 463837, > "max\_score" : 1.0, > "hits" : [ { > "\_index" : "graylog2\_recent", > "\_type" : "message", > "\_id" : "bfbebd30-b5db-11e2-aecd-00163e783829", > "\_score" : 1.0, > "fields" : { > "\_ttl" : -23976963, > "\_timestamp" : "06/May/2013:01:30:19 +0200" > } > }, { > "\_index" : "graylog2\_recent", > "\_type" : "message", > "\_id" : "bfcbb580-b5db-11e2-aecd-00163e783829", > "\_score" : 1.0, > "fields" : { > "\_ttl" : -23976878, > "\_timestamp" : "06/May/2013:01:30:19 +0200" > } > }, { > "\_index" : "graylog2\_recent", > "\_type" : "message", > "\_id" : "bfd37db0-b5db-11e2-aecd-00163e783829", > "\_score" : 1.0, > "fields" : { > "\_ttl" : -23976828 > } > }, { > "\_index" : "graylog2\_recent", > "\_type" : "message", > "\_id" : "bfeaaf30-b5db-11e2-aecd-00163e783829", > "\_score" : 1.0, > "fields" : { > "\_ttl" : -23976675, > "\_timestamp" : "06/May/2013:01:30:19 +0200" > } > }, { > [...] > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/RALBkPjLgac/unsubscribe?hl=en-US](https://groups.google.com/d/topic/elasticsearch/RALBkPjLgac/unsubscribe?hl=en-US). > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![brian\_yoder](https://avatars.discourse-cdn.com/v4/letter/b/f1d935/32.png) [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)\ **Post date:** [May 10, 2013, 4:12pm UTC](https://discuss.elastic.co/t/expired-documents-not-purged/11828/6 "2013-05-10T16:12:52Z")
You don't specify the records, but I do believe that \_ttl must be an attribute and not a field of the document. (My terminology). The \_ttl is specified during a bulk load as part of the action\_and\_metadata and not as a source field. The \_ttl is also able to be specified by the Java create and update APIs with other attributes such as index, type, and ID. I am re-running my TTL regression tests now from within 0.90.0 with Java 1.6 (yeah, I know.... beyond my control...) I queried and found (subset below): { "\_index" : "mortal" , "\_type" : "elf" , "\_id" : "1" , "\_version" : 1 , "\_score" : 1.0 , "\_source" : { "cn" : "Celeborn" , "text" : "Lives forever" } } { "\_index" : "mortal" , "\_type" : "elf" , "\_id" : "2" , "\_version" : 1 , "\_score" : 1.0 , "\_source" : { "cn" : "Galadriel" , "text" : "Lives forever" } } { "\_index" : "mortal" , "\_type" : "elf" , "\_id" : "3" , "\_version" : 1 , "\_score" : 1.0 , _"\_ttl" : "2.5m"_ , "\_source" : { "cn" : "Haldir of Lórien" , "text" : ["Dies in the battle of Helm's Deep" , "Gone in 3m"] } } Then that last one after a subsequent query: { "\_index" : "mortal" , "\_type" : "elf" , "\_id" : "3" , "\_version" : 1 , "\_score" : 1.0 , "\_ttl" : "-3624" , "\_source" : { "cn" : "Haldir of Lórien" , "text" : ["Dies in the battle of Helm's Deep" , "Gone in 3m"] } } And again: { "\_index" : "mortal" , "\_type" : "elf" , "\_id" : "3" , "\_version" : 1 , "\_score" : 1.0 , "\_ttl" : "-39484" , "\_source" : { "cn" : "Haldir of Lórien" , "text" : ["Dies in the battle of Helm's Deep" , "Gone in 3m"] } } And now here are the entire set of documents that remain at this moment: { "\_index" : "mortal" , "\_type" : "ghost" , "\_id" : "9" , "\_version" : 1 , "\_score" : 1.0 , "\_ttl" : "-18524" , "\_source" : { "cn" : "Boo Berry" , "text" : "Gone in 5 minutes" } } { "\_index" : "mortal" , "\_type" : "elf" , "\_id" : "1" , "\_version" : 1 , "\_score" : 1.0 , "\_source" : { "cn" : "Celeborn" , "text" : "Lives forever" } } { "\_index" : "mortal" , "\_type" : "elf" , "\_id" : "2" , "\_version" : 1 , "\_score" : 1.0 , "\_source" : { "cn" : "Galadriel" , "text" : "Lives forever" } } And waiting a little longer until the test run is completed, I'm left with the only 2 that are expected to remain: { "\_index" : "mortal" , "\_type" : "elf" , "\_id" : "1" , "\_version" : 1 , "\_score" : 1.0 , "\_source" : { "cn" : "Celeborn" , "text" : "Lives forever" } } { "\_index" : "mortal" , "\_type" : "elf" , "\_id" : "2" , "\_version" : 1 , "\_score" : 1.0 , "\_source" : { "cn" : "Galadriel" , "text" : "Lives forever" } } I don't remember if the expired documents are hanging around longer than they used to, but they do eventually all go away. On Friday, May 10, 2013 4:06:52 AM UTC-4, Stéphane Cottin wrote: > I was wrong, java update did not fix anything. > > ttl cleaning seems broken in 0.90. > running with debug log level show purge loops are fired, but they don't > delete anything. > > as a workaround, i'm running a cron job once per hour : > > #!/bin/sh > curl -XDELETE [http://127.0.0.1:9200/graylog2\_recent/\_query](http://127.0.0.1:9200/graylog2_recent/_query) -d > "{"range":{"\_ttl":{"to":$(date +%s)000}}}" > curl [http://127.0.0.1:9200/graylog2\_recent/\_optimize](http://127.0.0.1:9200/graylog2_recent/_optimize) > > not elegant, but works > > Le mercredi 8 mai 2013 10:04:15 UTC+2, Stéphane Cottin a écrit : > > > upgrade from openjdk 6 to openjdk 7 fix it. > > > > Le lundi 6 mai 2013 09:25:59 UTC+2, Stéphane Cottin a écrit : > > > > > Hi, > > > > > > I'm using ES 0.90 with graylog2, expired documents in recent index are > > > not purged. > > > appear ES related to me, is this a bug or a misconfiguration ? > > > how can i manually purge old entries ? > > > > > > thx > > > > > > \_settings : > > > > > > { > > > "graylog2\_recent" : { > > > "settings" : { > > > "index.store.type" : "niofs", > > > "index.number\_of\_replicas" : "0", > > > "index.number\_of\_shards" : "4", > > > "index.version.created" : "900099" > > > } > > > } > > > } > > > > > > \_mapping : > > > > > > { > > > "graylog2\_recent" : { > > > "message" : { > > > "dynamic\_templates" : [ { > > > "store\_generic" : { > > > "mapping" : { > > > "index" : "not\_analyzed" > > > }, > > > "match" : "\*" > > > } > > > } ], > > > "\_ttl" : { > > > "enabled" : true > > > }, > > > "\_source" : { > > > "compress" : true > > > }, > > > [...] > > > > > > [...]/graylog2\_recent/\_search?pretty=true&fields=\_ttl,\_timestamp > > > { > > > "took" : 8, > > > "timed\_out" : false, > > > "\_shards" : { > > > "total" : 4, > > > "successful" : 4, > > > "failed" : 0 > > > }, > > > "hits" : { > > > "total" : 463837, > > > "max\_score" : 1.0, > > > "hits" : [ { > > > "\_index" : "graylog2\_recent", > > > "\_type" : "message", > > > "\_id" : "bfbebd30-b5db-11e2-aecd-00163e783829", > > > "\_score" : 1.0, > > > "fields" : { > > > "\_ttl" : -23976963, > > > "\_timestamp" : "06/May/2013:01:30:19 +0200" > > > } > > > }, { > > > "\_index" : "graylog2\_recent", > > > "\_type" : "message", > > > "\_id" : "bfcbb580-b5db-11e2-aecd-00163e783829", > > > "\_score" : 1.0, > > > "fields" : { > > > "\_ttl" : -23976878, > > > "\_timestamp" : "06/May/2013:01:30:19 +0200" > > > } > > > }, { > > > "\_index" : "graylog2\_recent", > > > "\_type" : "message", > > > "\_id" : "bfd37db0-b5db-11e2-aecd-00163e783829", > > > "\_score" : 1.0, > > > "fields" : { > > > "\_ttl" : -23976828 > > > } > > > }, { > > > "\_index" : "graylog2\_recent", > > > "\_type" : "message", > > > "\_id" : "bfeaaf30-b5db-11e2-aecd-00163e783829", > > > "\_score" : 1.0, > > > "fields" : { > > > "\_ttl" : -23976675, > > > "\_timestamp" : "06/May/2013:01:30:19 +0200" > > > } > > > }, { > > > [...] -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Stephane\_Cottin](https://avatars.discourse-cdn.com/v4/letter/s/9de0a6/32.png) [@Stephane\_Cottin](https://discuss.elastic.co/u/Stephane_Cottin)\ **Post date:** [May 13, 2013, 8:51am UTC](https://discuss.elastic.co/t/expired-documents-not-purged/11828/7 "2013-05-13T08:51:52Z")
So ES is fine, the bug is in graylog2 , which either no longer use ttl in head dev version. I'll may stop using graylog2, but will still use ES in the future. Many thanks for running these tests. Le 10 mai 2013 à 18:12, InquiringMind [brian.from.fl@gmail.com](mailto:brian.from.fl@gmail.com) a écrit : > You don't specify the records, but I do believe that \_ttl must be an attribute and not a field of the document. (My terminology). The \_ttl is specified during a bulk load as part of the action\_and\_metadata and not as a source field. The \_ttl is also able to be specified by the Java create and update APIs with other attributes such as index, type, and ID. > > I am re-running my TTL regression tests now from within 0.90.0 with Java 1.6 (yeah, I know.... beyond my control...) > > I queried and found (subset below): > > { "\_index" : "mortal" , "\_type" : "elf" , "\_id" : "1" , "\_version" : 1 , "\_score" : 1.0 , "\_source" : { "cn" : "Celeborn" , "text" : "Lives forever" } } > > { "\_index" : "mortal" , "\_type" : "elf" , "\_id" : "2" , "\_version" : 1 , "\_score" : 1.0 , "\_source" : { "cn" : "Galadriel" , "text" : "Lives forever" } } > > { "\_index" : "mortal" , "\_type" : "elf" , "\_id" : "3" , "\_version" : 1 , "\_score" : 1.0 , "\_ttl" : "2.5m" , "\_source" : { "cn" : "Haldir of Lórien" , "text" : ["Dies in the battle of Helm's Deep" , "Gone in 3m"] } } > > Then that last one after a subsequent query: > > { "\_index" : "mortal" , "\_type" : "elf" , "\_id" : "3" , "\_version" : 1 , "\_score" : 1.0 , "\_ttl" : "-3624" , "\_source" : { "cn" : "Haldir of Lórien" , "text" : ["Dies in the battle of Helm's Deep" , "Gone in 3m"] } } > > And again: > > { "\_index" : "mortal" , "\_type" : "elf" , "\_id" : "3" , "\_version" : 1 , "\_score" : 1.0 , "\_ttl" : "-39484" , "\_source" : { "cn" : "Haldir of Lórien" , "text" : ["Dies in the battle of Helm's Deep" , "Gone in 3m"] } } > > And now here are the entire set of documents that remain at this moment: > > { "\_index" : "mortal" , "\_type" : "ghost" , "\_id" : "9" , "\_version" : 1 , "\_score" : 1.0 , "\_ttl" : "-18524" , "\_source" : { "cn" : "Boo Berry" , "text" : "Gone in 5 minutes" } } > { "\_index" : "mortal" , "\_type" : "elf" , "\_id" : "1" , "\_version" : 1 , "\_score" : 1.0 , "\_source" : { "cn" : "Celeborn" , "text" : "Lives forever" } } > { "\_index" : "mortal" , "\_type" : "elf" , "\_id" : "2" , "\_version" : 1 , "\_score" : 1.0 , "\_source" : { "cn" : "Galadriel" , "text" : "Lives forever" } } > > And waiting a little longer until the test run is completed, I'm left with the only 2 that are expected to remain: > > { "\_index" : "mortal" , "\_type" : "elf" , "\_id" : "1" , "\_version" : 1 , "\_score" : 1.0 , "\_source" : { "cn" : "Celeborn" , "text" : "Lives forever" } } > { "\_index" : "mortal" , "\_type" : "elf" , "\_id" : "2" , "\_version" : 1 , "\_score" : 1.0 , "\_source" : { "cn" : "Galadriel" , "text" : "Lives forever" } } > > I don't remember if the expired documents are hanging around longer than they used to, but they do eventually all go away. > > On Friday, May 10, 2013 4:06:52 AM UTC-4, Stéphane Cottin wrote: > I was wrong, java update did not fix anything. > > ttl cleaning seems broken in 0.90. > running with debug log level show purge loops are fired, but they don't delete anything. > > as a workaround, i'm running a cron job once per hour : > > #!/bin/sh > curl -XDELETE [http://127.0.0.1:9200/graylog2\_recent/\_query](http://127.0.0.1:9200/graylog2_recent/_query) -d "{"range":{"\_ttl":{"to":$(date +%s)000}}}" > curl [http://127.0.0.1:9200/graylog2\_recent/\_optimize](http://127.0.0.1:9200/graylog2_recent/_optimize) > > not elegant, but works > > Le mercredi 8 mai 2013 10:04:15 UTC+2, Stéphane Cottin a écrit : > upgrade from openjdk 6 to openjdk 7 fix it. > > Le lundi 6 mai 2013 09:25:59 UTC+2, Stéphane Cottin a écrit : > Hi, > > I'm using ES 0.90 with graylog2, expired documents in recent index are not purged. > appear ES related to me, is this a bug or a misconfiguration ? > how can i manually purge old entries ? > > thx > > \_settings : > > { > "graylog2\_recent" : { > "settings" : { > "index.store.type" : "niofs", > "index.number\_of\_replicas" : "0", > "index.number\_of\_shards" : "4", > "index.version.created" : "900099" > } > } > } > > \_mapping : > > { > "graylog2\_recent" : { > "message" : { > "dynamic\_templates" : [ { > "store\_generic" : { > "mapping" : { > "index" : "not\_analyzed" > }, > "match" : "\*" > } > } ], > "\_ttl" : { > "enabled" : true > }, > "\_source" : { > "compress" : true > }, > [...] > > [...]/graylog2\_recent/\_search?pretty=true&fields=\_ttl,\_timestamp > { > "took" : 8, > "timed\_out" : false, > "\_shards" : { > "total" : 4, > "successful" : 4, > "failed" : 0 > }, > "hits" : { > "total" : 463837, > "max\_score" : 1.0, > "hits" : [ { > "\_index" : "graylog2\_recent", > "\_type" : "message", > "\_id" : "bfbebd30-b5db-11e2-aecd-00163e783829", > "\_score" : 1.0, > "fields" : { > "\_ttl" : -23976963, > "\_timestamp" : "06/May/2013:01:30:19 +0200" > } > }, { > "\_index" : "graylog2\_recent", > "\_type" : "message", > "\_id" : "bfcbb580-b5db-11e2-aecd-00163e783829", > "\_score" : 1.0, > "fields" : { > "\_ttl" : -23976878, > "\_timestamp" : "06/May/2013:01:30:19 +0200" > } > }, { > "\_index" : "graylog2\_recent", > "\_type" : "message", > "\_id" : "bfd37db0-b5db-11e2-aecd-00163e783829", > "\_score" : 1.0, > "fields" : { > "\_ttl" : -23976828 > } > }, { > "\_index" : "graylog2\_recent", > "\_type" : "message", > "\_id" : "bfeaaf30-b5db-11e2-aecd-00163e783829", > "\_score" : 1.0, > "fields" : { > "\_ttl" : -23976675, > "\_timestamp" : "06/May/2013:01:30:19 +0200" > } > }, { > [...] > > -- > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/RALBkPjLgac/unsubscribe?hl=en-US](https://groups.google.com/d/topic/elasticsearch/RALBkPjLgac/unsubscribe?hl=en-US). > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:36am UTC](https://discuss.elastic.co/t/expired-documents-not-purged/11828/8 "2017-07-06T02:36:57Z")