TTL issue of ES 0.18.7

yangzhaoxia · June 4, 2012, 9:20am

My case is using logstash/rabbitmq/graylog2.
I have to use ES 0.18.7 to store messages for Graylog2 since ES 0.18.7
is required by logstash.
The index created by Graylog2 server is "graylog2". All the messages
are indexed by "graylog2".

Now I want to purge messages with 3 minutes expiration (3m is for
test), so I want to use TTL feature of index.
{code}
curl -XPOST 'localhost:9200/graylog2/_close'
{"ok":true,"acknowledged":true}

curl -XPUT http://localhost:9200/graylog2/_settings -d '{"graylog2":
{"_ttl" : { "enabled" : true, "default" : "3m"},"settings":
{"index.number_of_shards":"5","index.number_of_replicas":"1"}}}'

curl -XGET http://localhost:9200/graylog2/_settings
{"graylog2":{"settings":
{"index.number_of_shards":"5","index.number_of_replicas":"1","index.graylog2._ttl.enabled":"true","index.graylog2.settings.index.number_of_shards":"5","index.graylog2._ttl.default":"3d","index.graylog2.settings.index.number_of_replicas":"1"}}}

curl -XPOST 'localhost:9200/graylog2/_open'

{code}

I didn't see any messages deleted after 40 minutes'
observation(messages kept growing and never deleted from Graylog2 Web
GUI) but I thought new messages would be deleted once ttl over 3m.

Could you help me?
Besides, I set TTL for index level, does this mean the whole index
would be deleted after the ttl period, or the messages of the index
would be purged?

I also tried to set ttl according to
http://dev.nuclearrooster.com/2012/05/15/enable-elasticsearch-ttl-for-graylog2-messages/
but nothing changed.

-- check the mapping of graylog2

curl http://localhost:9200/graylog2/message/_mapping

{"message":{"dynamic_templates":[{"store_generic":{"mapping":
{"index":"not_analyzed"},"match":"*"}}],"properties":{"message":
{"analyzer":"whitespace","type":"string"},"level":
{"type":"long"},"host":
{"index":"not_analyzed","type":"string"},"facility":
{"index":"not_analyzed","type":"string"},"file":
{"index":"not_analyzed","type":"string"},"full_message":
{"analyzer":"whitespace","type":"string"},"line":
{"type":"long"},"created_at":{"type":"double"},"streams":
{"index":"not_analyzed","type":"string"}}}}[

-- add ttl setting

curl http://localhost:9200/graylog2/message/_mapping -XPUT -d

'{"message":{"dynamic_templates":[{"store_generic":{"mapping":
{"index":"not_analyzed"},"match":"*"}}],"_ttl" : { "enabled" : true,
"default" : "30d"},"properties":{"message":
{"analyzer":"whitespace","type":"string"},"level":
{"type":"long"},"host":
{"index":"not_analyzed","type":"string"},"facility":
{"index":"not_analyzed","type":"string"},"file":
{"index":"not_analyzed","type":"string"},"full_message":
{"analyzer":"whitespace","type":"string"},"line":
{"type":"long"},"created_at":{"type":"double"},"streams":
{"index":"not_analyzed","type":"string"}}}}'
{"ok":true,"acknowledged":true}

-- check mapping setting after adding ttl, but nothing changed

curl http://localhost:9200/graylog2/message/_mapping

{"message":{"dynamic_templates":[{"store_generic":{"mapping":
{"index":"not_analyzed"},"match":"*"}}],"properties":{"message":
{"analyzer":"whitespace","type":"string"},"level":
{"type":"long"},"host":
{"index":"not_analyzed","type":"string"},"facility":
{"index":"not_analyzed","type":"string"},"file":
{"index":"not_analyzed","type":"string"},"full_message":
{"analyzer":"whitespace","type":"string"},"line":
{"type":"long"},"created_at":{"type":"double"},"streams":
{"index":"not_analyzed","type":"string"}}}}

Regards,

Benjamin_Deveze · June 4, 2012, 9:36am

Heya,

if I remember well the dynamic mapping update of an index default
TTL was added after ES 0.18.7 that is certainly why you don't see
changes and why new indexed messages are not properly purged after 3
minutes
Only the messages of the index will be purged not the whole index

Hope this help

On Mon, Jun 4, 2012 at 11:20 AM, yang zhaoxia yangzhaoxia@gmail.com wrote:

My case is using logstash/rabbitmq/graylog2.
I have to use ES 0.18.7 to store messages for Graylog2 since ES 0.18.7
is required by logstash.
The index created by Graylog2 server is "graylog2". All the messages
are indexed by "graylog2".

Now I want to purge messages with 3 minutes expiration (3m is for
test), so I want to use TTL feature of index.
{code}
curl -XPOST 'localhost:9200/graylog2/_close'
{"ok":true,"acknowledged":true}

curl -XPUT http://localhost:9200/graylog2/_settings -d '{"graylog2":
{"_ttl" : { "enabled" : true, "default" : "3m"},"settings":
{"index.number_of_shards":"5","index.number_of_replicas":"1"}}}'

curl -XGET http://localhost:9200/graylog2/_settings
{"graylog2":{"settings":
{"index.number_of_shards":"5","index.number_of_replicas":"1","index.graylog2._ttl.enabled":"true","index.graylog2.settings.index.number_of_shards":"5","index.graylog2._ttl.default":"3d","index.graylog2.settings.index.number_of_replicas":"1"}}}

curl -XPOST 'localhost:9200/graylog2/_open'

{code}

I didn't see any messages deleted after 40 minutes'
observation(messages kept growing and never deleted from Graylog2 Web
GUI) but I thought new messages would be deleted once ttl over 3m.

Could you help me?
Besides, I set TTL for index level, does this mean the whole index
would be deleted after the ttl period, or the messages of the index
would be purged?

I also tried to set ttl according to
http://dev.nuclearrooster.com/2012/05/15/enable-elasticsearch-ttl-for-graylog2-messages/
but nothing changed.

-- check the mapping of graylog2

curl http://localhost:9200/graylog2/message/_mapping

{"message":{"dynamic_templates":[{"store_generic":{"mapping":
{"index":"not_analyzed"},"match":"*"}}],"properties":{"message":
{"analyzer":"whitespace","type":"string"},"level":
{"type":"long"},"host":
{"index":"not_analyzed","type":"string"},"facility":
{"index":"not_analyzed","type":"string"},"file":
{"index":"not_analyzed","type":"string"},"full_message":
{"analyzer":"whitespace","type":"string"},"line":
{"type":"long"},"created_at":{"type":"double"},"streams":
{"index":"not_analyzed","type":"string"}}}}[

-- add ttl setting

curl http://localhost:9200/graylog2/message/_mapping -XPUT -d

'{"message":{"dynamic_templates":[{"store_generic":{"mapping":
{"index":"not_analyzed"},"match":"*"}}],"_ttl" : { "enabled" : true,
"default" : "30d"},"properties":{"message":
{"analyzer":"whitespace","type":"string"},"level":
{"type":"long"},"host":
{"index":"not_analyzed","type":"string"},"facility":
{"index":"not_analyzed","type":"string"},"file":
{"index":"not_analyzed","type":"string"},"full_message":
{"analyzer":"whitespace","type":"string"},"line":
{"type":"long"},"created_at":{"type":"double"},"streams":
{"index":"not_analyzed","type":"string"}}}}'
{"ok":true,"acknowledged":true}

-- check mapping setting after adding ttl, but nothing changed

curl http://localhost:9200/graylog2/message/_mapping

{"message":{"dynamic_templates":[{"store_generic":{"mapping":
{"index":"not_analyzed"},"match":"*"}}],"properties":{"message":
{"analyzer":"whitespace","type":"string"},"level":
{"type":"long"},"host":
{"index":"not_analyzed","type":"string"},"facility":
{"index":"not_analyzed","type":"string"},"file":
{"index":"not_analyzed","type":"string"},"full_message":
{"analyzer":"whitespace","type":"string"},"line":
{"type":"long"},"created_at":{"type":"double"},"streams":
{"index":"not_analyzed","type":"string"}}}}

Regards,

--
Benjamin DEVEZE

yangzhaoxia · June 4, 2012, 9:49am

Thanks a lot, Benjamin.
At first I didn't run index close/open, then setting XPUT would cause
below warning in the ES server side:

[graylog2] ignoring non dynamic index level settings for open indices:
[index.graylog2._ttl.default, index.graylog2._ttl.enabled,
index.graylog2.settings.index.number_of_replicas,
index.graylog2.settings.index.number_of_shards]

From this warning, I thought it should support TTL changes after
running index close/open.
That's a pity.

On Jun 4, 5:36 pm, Benjamin Devèze benjamin.dev...@gmail.com wrote:

Heya,

if I remember well the dynamic mapping update of an index default
TTL was added after ES 0.18.7 that is certainly why you don't see
changes and why new indexed messages are not properly purged after 3
minutes

Only the messages of the index will be purged not the whole index

Hope this help

On Mon, Jun 4, 2012 at 11:20 AM, yang zhaoxia yangzhao...@gmail.com wrote:

My case is using logstash/rabbitmq/graylog2.
I have to use ES 0.18.7 to store messages for Graylog2 since ES 0.18.7
is required by logstash.
The index created by Graylog2 server is "graylog2". All the messages
are indexed by "graylog2".

Now I want to purge messages with 3 minutes expiration (3m is for
test), so I want to use TTL feature of index.
{code}
curl -XPOST 'localhost:9200/graylog2/_close'
{"ok":true,"acknowledged":true}

curl -XPUThttp://localhost:9200/graylog2/_settings-d '{"graylog2":
{"_ttl" : { "enabled" : true, "default" : "3m"},"settings":
{"index.number_of_shards":"5","index.number_of_replicas":"1"}}}'

curl -XGEThttp://localhost:9200/graylog2/_settings
{"graylog2":{"settings":
{"index.number_of_shards":"5","index.number_of_replicas":"1","index.graylog 2._ttl.enabled":"true","index.graylog2.settings.index.number_of_shards":"5" ,"index.graylog2._ttl.default":"3d","index.graylog2.settings.index.number_o f_replicas":"1"}}}

curl -XPOST 'localhost:9200/graylog2/_open'

{code}

I didn't see any messages deleted after 40 minutes'
observation(messages kept growing and never deleted from Graylog2 Web
GUI) but I thought new messages would be deleted once ttl over 3m.

Could you help me?
Besides, I set TTL for index level, does this mean the whole index
would be deleted after the ttl period, or the messages of the index
would be purged?

I also tried to set ttl according to
http://dev.nuclearrooster.com/2012/05/15/enable-elasticsearch-ttl-for...
but nothing changed.

-- check the mapping of graylog2

curlhttp://localhost:9200/graylog2/message/_mapping

{"message":{"dynamic_templates":[{"store_generic":{"mapping":
{"index":"not_analyzed"},"match":"*"}}],"properties":{"message":
{"analyzer":"whitespace","type":"string"},"level":
{"type":"long"},"host":
{"index":"not_analyzed","type":"string"},"facility":
{"index":"not_analyzed","type":"string"},"file":
{"index":"not_analyzed","type":"string"},"full_message":
{"analyzer":"whitespace","type":"string"},"line":
{"type":"long"},"created_at":{"type":"double"},"streams":
{"index":"not_analyzed","type":"string"}}}}[

-- add ttl setting

curlhttp://localhost:9200/graylog2/message/_mapping-XPUT -d

'{"message":{"dynamic_templates":[{"store_generic":{"mapping":
{"index":"not_analyzed"},"match":"*"}}],"_ttl" : { "enabled" : true,
"default" : "30d"},"properties":{"message":
{"analyzer":"whitespace","type":"string"},"level":
{"type":"long"},"host":
{"index":"not_analyzed","type":"string"},"facility":
{"index":"not_analyzed","type":"string"},"file":
{"index":"not_analyzed","type":"string"},"full_message":
{"analyzer":"whitespace","type":"string"},"line":
{"type":"long"},"created_at":{"type":"double"},"streams":
{"index":"not_analyzed","type":"string"}}}}'
{"ok":true,"acknowledged":true}

-- check mapping setting after adding ttl, but nothing changed

curlhttp://localhost:9200/graylog2/message/_mapping

{"message":{"dynamic_templates":[{"store_generic":{"mapping":
{"index":"not_analyzed"},"match":"*"}}],"properties":{"message":
{"analyzer":"whitespace","type":"string"},"level":
{"type":"long"},"host":
{"index":"not_analyzed","type":"string"},"facility":
{"index":"not_analyzed","type":"string"},"file":
{"index":"not_analyzed","type":"string"},"full_message":
{"analyzer":"whitespace","type":"string"},"line":
{"type":"long"},"created_at":{"type":"double"},"streams":
{"index":"not_analyzed","type":"string"}}}}

Regards,

--
Benjamin DEVEZE

yangzhaoxia · June 7, 2012, 9:14am

Since ES 0.18.7 doesn't supppor this TTL dynamic changes, I finally made it work as expected via Graylog2 Message Retention functionality.
Thanks.