So ES is fine, the bug is in graylog2 , which either no longer use ttl in head dev version.
I'll may stop using graylog2, but will still use ES in the future.
Many thanks for running these tests.
You don't specify the records, but I do believe that _ttl must be an attribute and not a field of the document. (My terminology). The _ttl is specified during a bulk load as part of the action_and_metadata and not as a source field. The _ttl is also able to be specified by the Java create and update APIs with other attributes such as index, type, and ID.
I am re-running my TTL regression tests now from within 0.90.0 with Java 1.6 (yeah, I know.... beyond my control...)
I queried and found (subset below):
{ "_index" : "mortal" , "_type" : "elf" , "_id" : "1" , "_version" : 1 , "_score" : 1.0 , "_source" : { "cn" : "Celeborn" , "text" : "Lives forever" } }
{ "_index" : "mortal" , "_type" : "elf" , "_id" : "2" , "_version" : 1 , "_score" : 1.0 , "_source" : { "cn" : "Galadriel" , "text" : "Lives forever" } }
{ "_index" : "mortal" , "_type" : "elf" , "_id" : "3" , "_version" : 1 , "_score" : 1.0 , "_ttl" : "2.5m" , "_source" : { "cn" : "Haldir of Lórien" , "text" : [ "Dies in the battle of Helm's Deep" , "Gone in 3m" ] } }
Then that last one after a subsequent query:
{ "_index" : "mortal" , "_type" : "elf" , "_id" : "3" , "_version" : 1 , "_score" : 1.0 , "_ttl" : "-3624" , "_source" : { "cn" : "Haldir of Lórien" , "text" : [ "Dies in the battle of Helm's Deep" , "Gone in 3m" ] } }
And again:
{ "_index" : "mortal" , "_type" : "elf" , "_id" : "3" , "_version" : 1 , "_score" : 1.0 , "_ttl" : "-39484" , "_source" : { "cn" : "Haldir of Lórien" , "text" : [ "Dies in the battle of Helm's Deep" , "Gone in 3m" ] } }
And now here are the entire set of documents that remain at this moment:
{ "_index" : "mortal" , "_type" : "ghost" , "_id" : "9" , "_version" : 1 , "_score" : 1.0 , "_ttl" : "-18524" , "_source" : { "cn" : "Boo Berry" , "text" : "Gone in 5 minutes" } }
{ "_index" : "mortal" , "_type" : "elf" , "_id" : "1" , "_version" : 1 , "_score" : 1.0 , "_source" : { "cn" : "Celeborn" , "text" : "Lives forever" } }
{ "_index" : "mortal" , "_type" : "elf" , "_id" : "2" , "_version" : 1 , "_score" : 1.0 , "_source" : { "cn" : "Galadriel" , "text" : "Lives forever" } }
And waiting a little longer until the test run is completed, I'm left with the only 2 that are expected to remain:
{ "_index" : "mortal" , "_type" : "elf" , "_id" : "1" , "_version" : 1 , "_score" : 1.0 , "_source" : { "cn" : "Celeborn" , "text" : "Lives forever" } }
{ "_index" : "mortal" , "_type" : "elf" , "_id" : "2" , "_version" : 1 , "_score" : 1.0 , "_source" : { "cn" : "Galadriel" , "text" : "Lives forever" } }
I don't remember if the expired documents are hanging around longer than they used to, but they do eventually all go away.
On Friday, May 10, 2013 4:06:52 AM UTC-4, Stéphane Cottin wrote:
I was wrong, java update did not fix anything.
ttl cleaning seems broken in 0.90.
running with debug log level show purge loops are fired, but they don't delete anything.
as a workaround, i'm running a cron job once per hour :
#!/bin/sh
curl -XDELETE http://127.0.0.1:9200/graylog2_recent/_query -d "{"range":{"_ttl":{"to":$(date +%s)000}}}"
curl http://127.0.0.1:9200/graylog2_recent/_optimize
not elegant, but works
Le mercredi 8 mai 2013 10:04:15 UTC+2, Stéphane Cottin a écrit :
upgrade from openjdk 6 to openjdk 7 fix it.
Le lundi 6 mai 2013 09:25:59 UTC+2, Stéphane Cottin a écrit :
Hi,
I'm using ES 0.90 with graylog2, expired documents in recent index are not purged.
appear ES related to me, is this a bug or a misconfiguration ?
how can i manually purge old entries ?
thx
_settings :
{
"graylog2_recent" : {
"settings" : {
"index.store.type" : "niofs",
"index.number_of_replicas" : "0",
"index.number_of_shards" : "4",
"index.version.created" : "900099"
}
}
}
_mapping :
{
"graylog2_recent" : {
"message" : {
"dynamic_templates" : [ {
"store_generic" : {
"mapping" : {
"index" : "not_analyzed"
},
"match" : "*"
}
} ],
"_ttl" : {
"enabled" : true
},
"_source" : {
"compress" : true
},
[...]
[...]/graylog2_recent/_search?pretty=true&fields=_ttl,_timestamp
{
"took" : 8,
"timed_out" : false,
"_shards" : {
"total" : 4,
"successful" : 4,
"failed" : 0
},
"hits" : {
"total" : 463837,
"max_score" : 1.0,
"hits" : [ {
"_index" : "graylog2_recent",
"_type" : "message",
"_id" : "bfbebd30-b5db-11e2-aecd-00163e783829",
"_score" : 1.0,
"fields" : {
"_ttl" : -23976963,
"_timestamp" : "06/May/2013:01:30:19 +0200"
}
}, {
"_index" : "graylog2_recent",
"_type" : "message",
"_id" : "bfcbb580-b5db-11e2-aecd-00163e783829",
"_score" : 1.0,
"fields" : {
"_ttl" : -23976878,
"_timestamp" : "06/May/2013:01:30:19 +0200"
}
}, {
"_index" : "graylog2_recent",
"_type" : "message",
"_id" : "bfd37db0-b5db-11e2-aecd-00163e783829",
"_score" : 1.0,
"fields" : {
"_ttl" : -23976828
}
}, {
"_index" : "graylog2_recent",
"_type" : "message",
"_id" : "bfeaaf30-b5db-11e2-aecd-00163e783829",
"_score" : 1.0,
"fields" : {
"_ttl" : -23976675,
"_timestamp" : "06/May/2013:01:30:19 +0200"
}
}, {
[...]
--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/RALBkPjLgac/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
