TTL issue of ES 0.18.7

Elastic Stack Elasticsearch
1

My case is using logstash/rabbitmq/graylog2.
I have to use ES 0.18.7 to store messages for Graylog2 since ES 0.18.7
is required by logstash.
The index created by Graylog2 server is "graylog2". All the messages
are indexed by "graylog2".

Now I want to purge messages with 3 minutes expiration (3m is for
test), so I want to use TTL feature of index.
{code}
curl -XPOST 'localhost:9200/graylog2/_close'
{"ok":true,"acknowledged":true}

curl -XPUT http://localhost:9200/graylog2/_settings -d '{"graylog2":
{"_ttl" : { "enabled" : true, "default" : "3m"},"settings":
{"index.number_of_shards":"5","index.number_of_replicas":"1"}}}'

curl -XGET http://localhost:9200/graylog2/_settings
{"graylog2":{"settings":
{"index.number_of_shards":"5","index.number_of_replicas":"1","index.graylog2._ttl.enabled":"true","index.graylog2.settings.index.number_of_shards":"5","index.graylog2._ttl.default":"3d","index.graylog2.settings.index.number_of_replicas":"1"}}}

curl -XPOST 'localhost:9200/graylog2/_open'

{code}

I didn't see any messages deleted after 40 minutes'
observation(messages kept growing and never deleted from Graylog2 Web
GUI) but I thought new messages would be deleted once ttl over 3m.

Could you help me?
Besides, I set TTL for index level, does this mean the whole index
would be deleted after the ttl period, or the messages of the index
would be purged?

I also tried to set ttl according to
http://dev.nuclearrooster.com/2012/05/15/enable-elasticsearch-ttl-for-graylog2-messages/
but nothing changed.

-- check the mapping of graylog2

curl http://localhost:9200/graylog2/message/_mapping

{"message":{"dynamic_templates":[{"store_generic":{"mapping":
{"index":"not_analyzed"},"match":"*"}}],"properties":{"message":
{"analyzer":"whitespace","type":"string"},"level":
{"type":"long"},"host":
{"index":"not_analyzed","type":"string"},"facility":
{"index":"not_analyzed","type":"string"},"file":
{"index":"not_analyzed","type":"string"},"full_message":
{"analyzer":"whitespace","type":"string"},"line":
{"type":"long"},"created_at":{"type":"double"},"streams":
{"index":"not_analyzed","type":"string"}}}}[

-- add ttl setting

curl http://localhost:9200/graylog2/message/_mapping -XPUT -d

'{"message":{"dynamic_templates":[{"store_generic":{"mapping":
{"index":"not_analyzed"},"match":"*"}}],"_ttl" : { "enabled" : true,
"default" : "30d"},"properties":{"message":
{"analyzer":"whitespace","type":"string"},"level":
{"type":"long"},"host":
{"index":"not_analyzed","type":"string"},"facility":
{"index":"not_analyzed","type":"string"},"file":
{"index":"not_analyzed","type":"string"},"full_message":
{"analyzer":"whitespace","type":"string"},"line":
{"type":"long"},"created_at":{"type":"double"},"streams":
{"index":"not_analyzed","type":"string"}}}}'
{"ok":true,"acknowledged":true}

-- check mapping setting after adding ttl, but nothing changed

curl http://localhost:9200/graylog2/message/_mapping

{"message":{"dynamic_templates":[{"store_generic":{"mapping":
{"index":"not_analyzed"},"match":"*"}}],"properties":{"message":
{"analyzer":"whitespace","type":"string"},"level":
{"type":"long"},"host":
{"index":"not_analyzed","type":"string"},"facility":
{"index":"not_analyzed","type":"string"},"file":
{"index":"not_analyzed","type":"string"},"full_message":
{"analyzer":"whitespace","type":"string"},"line":
{"type":"long"},"created_at":{"type":"double"},"streams":
{"index":"not_analyzed","type":"string"}}}}

Regards,

2

Heya,

  1. if I remember well the dynamic mapping update of an index default
    TTL was added after ES 0.18.7 that is certainly why you don't see
    changes and why new indexed messages are not properly purged after 3
    minutes
  2. Only the messages of the index will be purged not the whole index

Hope this help

On Mon, Jun 4, 2012 at 11:20 AM, yang zhaoxia yangzhaoxia@gmail.com wrote:

My case is using logstash/rabbitmq/graylog2.
I have to use ES 0.18.7 to store messages for Graylog2 since ES 0.18.7
is required by logstash.
The index created by Graylog2 server is "graylog2". All the messages
are indexed by "graylog2".

Now I want to purge messages with 3 minutes expiration (3m is for
test), so I want to use TTL feature of index.
{code}
curl -XPOST 'localhost:9200/graylog2/_close'
{"ok":true,"acknowledged":true}

curl -XPUT http://localhost:9200/graylog2/_settings -d '{"graylog2":
{"_ttl" : { "enabled" : true, "default" : "3m"},"settings":
{"index.number_of_shards":"5","index.number_of_replicas":"1"}}}'

curl -XGET http://localhost:9200/graylog2/_settings
{"graylog2":{"settings":
{"index.number_of_shards":"5","index.number_of_replicas":"1","index.graylog2._ttl.enabled":"true","index.graylog2.settings.index.number_of_shards":"5","index.graylog2._ttl.default":"3d","index.graylog2.settings.index.number_of_replicas":"1"}}}

curl -XPOST 'localhost:9200/graylog2/_open'

{code}

I didn't see any messages deleted after 40 minutes'
observation(messages kept growing and never deleted from Graylog2 Web
GUI) but I thought new messages would be deleted once ttl over 3m.

Could you help me?
Besides, I set TTL for index level, does this mean the whole index
would be deleted after the ttl period, or the messages of the index
would be purged?

I also tried to set ttl according to
http://dev.nuclearrooster.com/2012/05/15/enable-elasticsearch-ttl-for-graylog2-messages/
but nothing changed.

-- check the mapping of graylog2

curl http://localhost:9200/graylog2/message/_mapping

{"message":{"dynamic_templates":[{"store_generic":{"mapping":
{"index":"not_analyzed"},"match":"*"}}],"properties":{"message":
{"analyzer":"whitespace","type":"string"},"level":
{"type":"long"},"host":
{"index":"not_analyzed","type":"string"},"facility":
{"index":"not_analyzed","type":"string"},"file":
{"index":"not_analyzed","type":"string"},"full_message":
{"analyzer":"whitespace","type":"string"},"line":
{"type":"long"},"created_at":{"type":"double"},"streams":
{"index":"not_analyzed","type":"string"}}}}[

-- add ttl setting

curl http://localhost:9200/graylog2/message/_mapping -XPUT -d

'{"message":{"dynamic_templates":[{"store_generic":{"mapping":
{"index":"not_analyzed"},"match":"*"}}],"_ttl" : { "enabled" : true,
"default" : "30d"},"properties":{"message":
{"analyzer":"whitespace","type":"string"},"level":
{"type":"long"},"host":
{"index":"not_analyzed","type":"string"},"facility":
{"index":"not_analyzed","type":"string"},"file":
{"index":"not_analyzed","type":"string"},"full_message":
{"analyzer":"whitespace","type":"string"},"line":
{"type":"long"},"created_at":{"type":"double"},"streams":
{"index":"not_analyzed","type":"string"}}}}'
{"ok":true,"acknowledged":true}

-- check mapping setting after adding ttl, but nothing changed

curl http://localhost:9200/graylog2/message/_mapping

{"message":{"dynamic_templates":[{"store_generic":{"mapping":
{"index":"not_analyzed"},"match":"*"}}],"properties":{"message":
{"analyzer":"whitespace","type":"string"},"level":
{"type":"long"},"host":
{"index":"not_analyzed","type":"string"},"facility":
{"index":"not_analyzed","type":"string"},"file":
{"index":"not_analyzed","type":"string"},"full_message":
{"analyzer":"whitespace","type":"string"},"line":
{"type":"long"},"created_at":{"type":"double"},"streams":
{"index":"not_analyzed","type":"string"}}}}

Regards,

--
Benjamin DEVEZE

3

Thanks a lot, Benjamin.
At first I didn't run index close/open, then setting XPUT would cause
below warning in the ES server side:

[graylog2] ignoring non dynamic index level settings for open indices:
[index.graylog2._ttl.default, index.graylog2._ttl.enabled,
index.graylog2.settings.index.number_of_replicas,
index.graylog2.settings.index.number_of_shards]

From this warning, I thought it should support TTL changes after
running index close/open.
That's a pity.

On Jun 4, 5:36 pm, Benjamin Devèze benjamin.dev...@gmail.com wrote:

Heya,

  1. if I remember well the dynamic mapping update of an index default
    TTL was added after ES 0.18.7 that is certainly why you don't see
    changes and why new indexed messages are not properly purged after 3
    minutes
  2. Only the messages of the index will be purged not the whole index

Hope this help

On Mon, Jun 4, 2012 at 11:20 AM, yang zhaoxia yangzhao...@gmail.com wrote:

My case is using logstash/rabbitmq/graylog2.
I have to use ES 0.18.7 to store messages for Graylog2 since ES 0.18.7
is required by logstash.
The index created by Graylog2 server is "graylog2". All the messages
are indexed by "graylog2".

Now I want to purge messages with 3 minutes expiration (3m is for
test), so I want to use TTL feature of index.
{code}
curl -XPOST 'localhost:9200/graylog2/_close'
{"ok":true,"acknowledged":true}

curl -XPUThttp://localhost:9200/graylog2/_settings-d '{"graylog2":
{"_ttl" : { "enabled" : true, "default" : "3m"},"settings":
{"index.number_of_shards":"5","index.number_of_replicas":"1"}}}'

curl -XGEThttp://localhost:9200/graylog2/_settings
{"graylog2":{"settings":
{"index.number_of_shards":"5","index.number_of_replicas":"1","index.graylog 2._ttl.enabled":"true","index.graylog2.settings.index.number_of_shards":"5" ,"index.graylog2._ttl.default":"3d","index.graylog2.settings.index.number_o f_replicas":"1"}}}

curl -XPOST 'localhost:9200/graylog2/_open'

{code}

I didn't see any messages deleted after 40 minutes'
observation(messages kept growing and never deleted from Graylog2 Web
GUI) but I thought new messages would be deleted once ttl over 3m.

Could you help me?
Besides, I set TTL for index level, does this mean the whole index
would be deleted after the ttl period, or the messages of the index
would be purged?

I also tried to set ttl according to
http://dev.nuclearrooster.com/2012/05/15/enable-elasticsearch-ttl-for...
but nothing changed.

-- check the mapping of graylog2

curlhttp://localhost:9200/graylog2/message/_mapping

{"message":{"dynamic_templates":[{"store_generic":{"mapping":
{"index":"not_analyzed"},"match":"*"}}],"properties":{"message":
{"analyzer":"whitespace","type":"string"},"level":
{"type":"long"},"host":
{"index":"not_analyzed","type":"string"},"facility":
{"index":"not_analyzed","type":"string"},"file":
{"index":"not_analyzed","type":"string"},"full_message":
{"analyzer":"whitespace","type":"string"},"line":
{"type":"long"},"created_at":{"type":"double"},"streams":
{"index":"not_analyzed","type":"string"}}}}[

-- add ttl setting

curlhttp://localhost:9200/graylog2/message/_mapping-XPUT -d

'{"message":{"dynamic_templates":[{"store_generic":{"mapping":
{"index":"not_analyzed"},"match":"*"}}],"_ttl" : { "enabled" : true,
"default" : "30d"},"properties":{"message":
{"analyzer":"whitespace","type":"string"},"level":
{"type":"long"},"host":
{"index":"not_analyzed","type":"string"},"facility":
{"index":"not_analyzed","type":"string"},"file":
{"index":"not_analyzed","type":"string"},"full_message":
{"analyzer":"whitespace","type":"string"},"line":
{"type":"long"},"created_at":{"type":"double"},"streams":
{"index":"not_analyzed","type":"string"}}}}'
{"ok":true,"acknowledged":true}

-- check mapping setting after adding ttl, but nothing changed

curlhttp://localhost:9200/graylog2/message/_mapping

{"message":{"dynamic_templates":[{"store_generic":{"mapping":
{"index":"not_analyzed"},"match":"*"}}],"properties":{"message":
{"analyzer":"whitespace","type":"string"},"level":
{"type":"long"},"host":
{"index":"not_analyzed","type":"string"},"facility":
{"index":"not_analyzed","type":"string"},"file":
{"index":"not_analyzed","type":"string"},"full_message":
{"analyzer":"whitespace","type":"string"},"line":
{"type":"long"},"created_at":{"type":"double"},"streams":
{"index":"not_analyzed","type":"string"}}}}

Regards,

--
Benjamin DEVEZE

4

Since ES 0.18.7 doesn't supppor this TTL dynamic changes, I finally made it work as expected via Graylog2 Message Retention functionality.
Thanks.