TTL set and enabled but logs not expiring ...please help

Elastic Stack Elasticsearch
1

I have TTL enabled, but the logs are not removed when I view them from
Kibana. They still show up, even after their expiration time is over.
Here is the mapping that I get from the API, which clearly shows that the
TTL is enabled with a default value of 1 day:

{"logstash-2013.05.14":{"default":{"_ttl":{"enabled":true,"default":86400000},"properties":{}},"filetype":{"_ttl":{"enabled":true,"default":86400000},"properties":{"@fields":{"properties":{"_second":{"type":"string"},"fiber1":{"type":"string"},"fiber2":{"type":"string"},"hour":{"type":"string"},"message":{"type":"string"},"minute":{"type":"string"},"monthday":{"type":"string"},"monthnum":{"type":"string"},"second":{"type":"string"},"slave":{"type":"string"},"ts":{"type":"string"},"type1":{"type":"string"},"type2":{"type":"string"},"year":{"type":"string"}}},"@message":{"type":"string"},"@source":{"type":"string"},"@source_host":{"type":"string"},"@source_path":{"type":"string"},"@tags":{"type":"string"},"@timestamp":{"type":"date","format":"dateOptionalTime"},"@type":{"type":"string"}}}},"logstash-2013.05.09":{"default":{"_ttl":{"enabled":true,"default":86400000},"properties":{}},"filetype":{"_ttl":{"enabled":true,"default":86400000},"properties":{"@fields":{"properties":{"_second":{"type":"string"},"fiber1":{"type":"string"},"fiber2":{"type":"string"},"hour":{"type":"string"},"message":{"type":"string"},"minute":{"type":"string"},"monthday":{"type":"string"},"monthnum":{"type":"string"},"second":{"type":"string"},"slave":{"type":"string"},"ts":{"type":"string"},"type1":{"type":"string"},"type2":{"type":"string"},"year":{"type":"string"}}},"@message":{"type":"string"},"@source":{"type":"string"},"@source_host":{"type":"string"},"@source_path":{"type":"string"},"@tags":{"type":"string"},"@timestamp":{"type":"date","format":"dateOptionalTime"},"@type":{"type":"string"}}}}}

Where am I going wrong? I am badly stuck here :frowning:

Please give valuable suggestions and help

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Aakash - please stop sending the same emails repeatedly

Send the question, and wait for a response. If you don't get one after a
few days, perhaps you need to clarify the question

clint

On 15 May 2013 15:17, Aakash Anuj aakashanuj.iitkgp@gmail.com wrote:

I have TTL enabled, but the logs are not removed when I view them from
Kibana. They still show up, even after their expiration time is over.
Here is the mapping that I get from the API, which clearly shows that the
TTL is enabled with a default value of 1 day:

{"logstash-2013.05.14":{"**default":{"ttl":{"enabled":**true,"default":86400000},"**properties":{}},"filetype":{"ttl":{"enabled":true,"default":86400000},"properties":{"@**fields":{"properties":{"**second":{"type":"string"},"**fiber1":{"type":"string"},"**fiber2":{"type":"string"},"**hour":{"type":"string"},"**message":{"type":"string"},"**minute":{"type":"string"},"**monthday":{"type":"string"},"**monthnum":{"type":"string"},"**second":{"type":"string"},"slave":{"type":"string"},"ts":{"type":"string"},"type1":{"**type":"string"},"type2":{"type":"string"},"year":{"type":"string"}}},"@message":{"**type":"string"},"@source":{"type":"string"},"@source_host":{"type":"string"},"@source**path":{"type":"string"},"@**tags":{"type":"string"},"@**timestamp":{"type":"date","**format":"dateOptionalTime"},"@**type":{"type":"string"}}}},"**logstash-2013.05.09":{"**default":{"ttl":{"enabled":**true,"default":86400000},"**properties":{}},"filetype":{"ttl":{"enabled":true,"default":86400000},"properties":{"@**fields":{"properties":{"**second":{"type":"string"},"**fiber1":{"type":"string"},"**fiber2":{"type":"string"},"**hour":{"type":"string"},"**message":{"type":"string"},"**minute":{"type":"string"},"**monthday":{"type":"string"},"**monthnum":{"type":"string"},"**second":{"type":"string"},"slave":{"type":"string"},"ts":{"type":"string"},"type1":{"**type":"string"},"type2":{"type":"string"},"year":{"type":"string"}}},"@message":{"**type":"string"},"@source":{"type":"string"},"@source_host":{"type":"string"},"@source**path":{"type":"string"},"@**tags":{"type":"string"},"@**timestamp":{"type":"date","**format":"dateOptionalTime"},"@**type":{"type":"string"}}}}}

Where am I going wrong? I am badly stuck here :frowning:

Please give valuable suggestions and help

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

It's kind of urgent and there is no reply... :frowning:

On Wednesday, May 15, 2013, Clinton Gormley wrote:

Aakash - please stop sending the same emails repeatedly

Send the question, and wait for a response. If you don't get one after a
few days, perhaps you need to clarify the question

clint

On 15 May 2013 15:17, Aakash Anuj <aakashanuj.iitkgp@gmail.com<javascript:_e({}, 'cvml', 'aakashanuj.iitkgp@gmail.com');>

wrote:

I have TTL enabled, but the logs are not removed when I view them from
Kibana. They still show up, even after their expiration time is over.
Here is the mapping that I get from the API, which clearly shows that the
TTL is enabled with a default value of 1 day:

{"logstash-2013.05.14":{"**default":{"ttl":{"enabled":**true,"default":86400000},"**properties":{}},"filetype":{"ttl":{"enabled":true,"default":86400000},"properties":{"@**fields":{"properties":{"**second":{"type":"string"},"**fiber1":{"type":"string"},"**fiber2":{"type":"string"},"**hour":{"type":"string"},"**message":{"type":"string"},"**minute":{"type":"string"},"**monthday":{"type":"string"},"**monthnum":{"type":"string"},"**second":{"type":"string"},"slave":{"type":"string"},"ts":{"type":"string"},"type1":{"**type":"string"},"type2":{"type":"string"},"year":{"type":"string"}}},"@message":{"**type":"string"},"@source":{"type":"string"},"@source_host":{"type":"string"},"@source**path":{"type":"string"},"@**tags":{"type":"string"},"@**timestamp":{"type":"date","**format":"dateOptionalTime"},"@**type":{"type":"string"}}}},"**logstash-2013.05.09":{"**default":{"ttl":{"enabled":**true,"default":86400000},"**properties":{}},"filetype":{"ttl":{"enabled":true,"default":86400000},"properties":{"@**fields":{"properties":{"**second":{"type":"string"},"**fiber1":{"type":"string"},"**fiber2":{"type":"string"},"**hour":{"type":"string"},"**message":{"type":"string"},"**minute":{"type":"string"},"**monthday":{"type":"string"},"**monthnum":{"type":"string"},"**second":{"type":"string"},"slave":{"type":"string"},"ts":{"type":"string"},"type1":{"**type":"string"},"type2":{"type":"string"},"year":{"type":"string"}}},"@message":{"**type":"string"},"@source":{"type":"string"},"@source_host":{"type":"string"},"@source**path":{"type":"string"},"@**tags":{"type":"string"},"@**timestamp":{"type":"date","**format":"dateOptionalTime"},"@**type":{"type":"string"}}}}}

Where am I going wrong? I am badly stuck here :frowning:

Please give valuable suggestions and help

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com <javascript:_e({},
'cvml', 'elasticsearch%2Bunsubscribe@googlegroups.com');>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/3j77O3pOCRc/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com <javascript:_e({}, 'cvml',
'elasticsearch%2Bunsubscribe@googlegroups.com');>.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Then you have three choices:

  1. Wait for a reply
  2. Read the documentation
  3. Pay for support

But resending emails is considered spam.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

I understand.
But had the doc been written that well, I would not have been asking this
question :stuck_out_tongue:

On Wednesday, May 15, 2013, Clinton Gormley wrote:

Then you have three choices:

  1. Wait for a reply
  2. Read the documentation
  3. Pay for support

But resending emails is considered spam.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/3j77O3pOCRc/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com <javascript:_e({}, 'cvml',
'elasticsearch%2Bunsubscribe@googlegroups.com');>.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

I have tested out your settings and they work. Which makes me think that
either:

  1. you are setting a ttl when you index each doc
  2. you have disabled expiry
  3. you only enabled default expiry after already indexing docs (it doesn't
    affect existing docs)

Any of these true?

But as David pointed out, it is much much more efficient to have an index
per day, and drop the old indices when you no longer need them

On 15 May 2013 15:52, Aakash Anuj aakashanuj.iitkgp@gmail.com wrote:

I understand.
But had the doc been written that well, I would not have been asking this
question :stuck_out_tongue:

On Wednesday, May 15, 2013, Clinton Gormley wrote:

Then you have three choices:

  1. Wait for a reply
  2. Read the documentation
  3. Pay for support

But resending emails is considered spam.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/3j77O3pOCRc/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

Hey,

All I have done is this:

I have created a folder mappings/_default/default.json which contains the
ttl enabling with the TTL duration. Since I am shipping my logs with
logstash, there will be a index per day.
What I know is that mappings/_default/default.json will apply the mapping
to all new indices created.

Do I need to do anything else? They above would enable the TTL, ryt?

On Wed, May 15, 2013 at 7:59 PM, Clinton Gormley clint@traveljury.comwrote:

I have tested out your settings and they work. Which makes me think that
either:

  1. you are setting a ttl when you index each doc
  2. you have disabled expiry
  3. you only enabled default expiry after already indexing docs (it doesn't
    affect existing docs)

Any of these true?

But as David pointed out, it is much much more efficient to have an index
per day, and drop the old indices when you no longer need them

On 15 May 2013 15:52, Aakash Anuj aakashanuj.iitkgp@gmail.com wrote:

I understand.
But had the doc been written that well, I would not have been asking this
question :stuck_out_tongue:

On Wednesday, May 15, 2013, Clinton Gormley wrote:

Then you have three choices:

  1. Wait for a reply
  2. Read the documentation
  3. Pay for support

But resending emails is considered spam.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/3j77O3pOCRc/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/3j77O3pOCRc/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

8

I don't know why this isn't working for you. It certainly works for me.

But if you're creating an index per day, then you don't need to worry about
ttl. You just drop the old indices.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

9

But how to I drop the indices which are old automatically from the ELASTIC
SEARCH database?

And please explain me one more thing.....
When is the file defaul.json loaded, that is, when is the TTL enabled? Is
it when Elasticsearch runs?

Also, how do I actually know if the indices get deleted?
Even if I add logs which have a TTL of 1 day and a timestamp sometime
before 1 day, they get added to ES. Why ?

I sorry for so many questions, but its really getting on my nerves now :frowning:

On Wed, May 15, 2013 at 8:20 PM, Clinton Gormley clint@traveljury.comwrote:

I don't know why this isn't working for you. It certainly works for me.

But if you're creating an index per day, then you don't need to worry
about ttl. You just drop the old indices.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/3j77O3pOCRc/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

10

I am sorry*

On Wed, May 15, 2013 at 8:23 PM, Aakash Anuj aakashanuj.iitkgp@gmail.comwrote:

But how to I drop the indices which are old automatically from the ELASTIC
SEARCH database?

And please explain me one more thing.....
When is the file defaul.json loaded, that is, when is the TTL enabled?
Is it when Elasticsearch runs?

Also, how do I actually know if the indices get deleted?
Even if I add logs which have a TTL of 1 day and a timestamp sometime
before 1 day, they get added to ES. Why ?

I sorry for so many questions, but its really getting on my nerves now :frowning:

On Wed, May 15, 2013 at 8:20 PM, Clinton Gormley clint@traveljury.comwrote:

I don't know why this isn't working for you. It certainly works for me.

But if you're creating an index per day, then you don't need to worry
about ttl. You just drop the old indices.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/3j77O3pOCRc/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

11

On 15 May 2013 16:53, Aakash Anuj aakashanuj.iitkgp@gmail.com wrote:

But how to I drop the indices which are old automatically from the ELASTIC
SEARCH database?

you don't. you use a cron daemon to run a job once a day

And please explain me one more thing.....
When is the file defaul.json loaded, that is, when is the TTL enabled?
Is it when Elasticsearch runs?

Only when you restart a node. And its values only apply when you create a
new index

Also, how do I actually know if the indices get deleted?

You make your cron job send you an email

Even if I add logs which have a TTL of 1 day and a timestamp sometime
before 1 day, they get added to ES. Why ?

The ttl depends on the time you index it, not the timestamp field.

This could explain why things weren't working the way you expected. Your
expectations were incorrect.

Please read this advice about how to ask questions in the best way to get
answers: Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

12

Oh, I guess the last point was where I was fumbling.

I had read on a blog somewhere that the TTL applies relative to the
timestamp. So do you mean that the TTL applies relative to the system time
of creation?

On Wed, May 15, 2013 at 8:55 PM, Clinton Gormley clint@traveljury.comwrote:

On 15 May 2013 16:53, Aakash Anuj aakashanuj.iitkgp@gmail.com wrote:

But how to I drop the indices which are old automatically from the
ELASTIC SEARCH database?

you don't. you use a cron daemon to run a job once a day

And please explain me one more thing.....
When is the file defaul.json loaded, that is, when is the TTL enabled?
Is it when Elasticsearch runs?

Only when you restart a node. And its values only apply when you create
a new index

Also, how do I actually know if the indices get deleted?

You make your cron job send you an email

Even if I add logs which have a TTL of 1 day and a timestamp sometime
before 1 day, they get added to ES. Why ?

The ttl depends on the time you index it, not the timestamp field.

This could explain why things weren't working the way you expected. Your
expectations were incorrect.

Please read this advice about how to ask questions in the best way to get
answers: Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/3j77O3pOCRc/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

13

I've just asked, and apparently I'm incorrect. ttl and timestamp ARE
related.

But: i see a "@timestamp" field in your mapping, but no _timestamp path
mapping to point to that field

clint

On 15 May 2013 17:32, Aakash Anuj aakashanuj.iitkgp@gmail.com wrote:

Oh, I guess the last point was where I was fumbling.

I had read on a blog somewhere that the TTL applies relative to the
timestamp. So do you mean that the TTL applies relative to the system time
of creation?

On Wed, May 15, 2013 at 8:55 PM, Clinton Gormley clint@traveljury.comwrote:

On 15 May 2013 16:53, Aakash Anuj aakashanuj.iitkgp@gmail.com wrote:

But how to I drop the indices which are old automatically from the
ELASTIC SEARCH database?

you don't. you use a cron daemon to run a job once a day

And please explain me one more thing.....
When is the file defaul.json loaded, that is, when is the TTL enabled?
Is it when Elasticsearch runs?

Only when you restart a node. And its values only apply when you create
a new index

Also, how do I actually know if the indices get deleted?

You make your cron job send you an email

Even if I add logs which have a TTL of 1 day and a timestamp sometime
before 1 day, they get added to ES. Why ?

The ttl depends on the time you index it, not the timestamp field.

This could explain why things weren't working the way you expected. Your
expectations were incorrect.

Please read this advice about how to ask questions in the best way to get
answers: Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/3j77O3pOCRc/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

14

You were right I guess....becoz now my logs are getting deleted after the
specified time after the indexing, not the time stamp value. Any idea how
to automatically delete the logs relative to the time stamp value?

On Wednesday, May 15, 2013, Clinton Gormley wrote:

I've just asked, and apparently I'm incorrect. ttl and timestamp ARE
related.

But: i see a "@timestamp" field in your mapping, but no _timestamp path
mapping to point to that field
Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

On 15 May 2013 17:32, Aakash Anuj <aakashanuj.iitkgp@gmail.com<javascript:_e({}, 'cvml', 'aakashanuj.iitkgp@gmail.com');>

wrote:

Oh, I guess the last point was where I was fumbling.

I had read on a blog somewhere that the TTL applies relative to the
timestamp. So do you mean that the TTL applies relative to the system time
of creation?

On Wed, May 15, 2013 at 8:55 PM, Clinton Gormley <clint@traveljury.com<javascript:_e({}, 'cvml', 'clint@traveljury.com');>

wrote:

On 15 May 2013 16:53, Aakash Anuj <aakashanuj.iitkgp@gmail.com<javascript:_e({}, 'cvml', 'aakashanuj.iitkgp@gmail.com');>

wrote:

But how to I drop the indices which are old automatically from the
ELASTIC SEARCH database?

you don't. you use a cron daemon to run a job once a day

And please explain me one more thing.....
When is the file defaul.json loaded, that is, when is the TTL
enabled? Is it when Elasticsearch runs?

Only when you restart a node. And its values only apply when you
create a new index

Also, how do I actually know if the indices get deleted?

You make your cron job send you an email

Even if I add logs which have a TTL of 1 day and a timestamp sometime
before 1 day, they get added to ES. Why ?

The ttl depends on the time you index it, not the timestamp field.

This could explain why things weren't working the way you expected.
Your expectations were incorrect.

Please read this advice about how to ask questions in the best way to
get answers: Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/3j77O3pOCRc/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com <javascript:_e({}, 'cvml',
'elasticsearch%2Bunsubscribe@googlegroups.com');>.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com <javascript:_e({},
'cvml', 'elasticsearch%2Bunsubscribe@googlegroups.com');>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/3j77O3pOCRc/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com <javascript:_e({}, 'cvml',
'elasticsearch%2Bunsubscribe@googlegroups.com');>.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

15

Could you help me with this?

https://groups.google.com/forum/?fromgroups=#!topic/elasticsearch/2jUyZY8B30Q

On Wed, May 15, 2013 at 11:47 PM, Aakash Anuj
aakashanuj.iitkgp@gmail.comwrote:

You were right I guess....becoz now my logs are getting deleted after the
specified time after the indexing, not the time stamp value. Any idea how
to automatically delete the logs relative to the time stamp value?

On Wednesday, May 15, 2013, Clinton Gormley wrote:

I've just asked, and apparently I'm incorrect. ttl and timestamp ARE
related.

But: i see a "@timestamp" field in your mapping, but no _timestamp path
mapping to point to that field
Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

On 15 May 2013 17:32, Aakash Anuj aakashanuj.iitkgp@gmail.com wrote:

Oh, I guess the last point was where I was fumbling.

I had read on a blog somewhere that the TTL applies relative to the
timestamp. So do you mean that the TTL applies relative to the system time
of creation?

On Wed, May 15, 2013 at 8:55 PM, Clinton Gormley clint@traveljury.comwrote:

On 15 May 2013 16:53, Aakash Anuj aakashanuj.iitkgp@gmail.com wrote:

But how to I drop the indices which are old automatically from the
ELASTIC SEARCH database?

you don't. you use a cron daemon to run a job once a day

And please explain me one more thing.....
When is the file defaul.json loaded, that is, when is the TTL
enabled? Is it when Elasticsearch runs?

Only when you restart a node. And its values only apply when you
create a new index

Also, how do I actually know if the indices get deleted?

You make your cron job send you an email

Even if I add logs which have a TTL of 1 day and a timestamp sometime
before 1 day, they get added to ES. Why ?

The ttl depends on the time you index it, not the timestamp field.

This could explain why things weren't working the way you expected.
Your expectations were incorrect.

Please read this advice about how to ask questions in the best way to
get answers: Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/3j77O3pOCRc/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/3j77O3pOCRc/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

16

Use delete-by-query

On 15 May 2013 20:17, Aakash Anuj aakashanuj.iitkgp@gmail.com wrote:

You were right I guess....becoz now my logs are getting deleted after the
specified time after the indexing, not the time stamp value. Any idea how
to automatically delete the logs relative to the time stamp value?

On Wednesday, May 15, 2013, Clinton Gormley wrote:

I've just asked, and apparently I'm incorrect. ttl and timestamp ARE
related.

But: i see a "@timestamp" field in your mapping, but no _timestamp path
mapping to point to that field
Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

On 15 May 2013 17:32, Aakash Anuj aakashanuj.iitkgp@gmail.com wrote:

Oh, I guess the last point was where I was fumbling.

I had read on a blog somewhere that the TTL applies relative to the
timestamp. So do you mean that the TTL applies relative to the system time
of creation?

On Wed, May 15, 2013 at 8:55 PM, Clinton Gormley clint@traveljury.comwrote:

On 15 May 2013 16:53, Aakash Anuj aakashanuj.iitkgp@gmail.com wrote:

But how to I drop the indices which are old automatically from the
ELASTIC SEARCH database?

you don't. you use a cron daemon to run a job once a day

And please explain me one more thing.....
When is the file defaul.json loaded, that is, when is the TTL
enabled? Is it when Elasticsearch runs?

Only when you restart a node. And its values only apply when you
create a new index

Also, how do I actually know if the indices get deleted?

You make your cron job send you an email

Even if I add logs which have a TTL of 1 day and a timestamp sometime
before 1 day, they get added to ES. Why ?

The ttl depends on the time you index it, not the timestamp field.

This could explain why things weren't working the way you expected.
Your expectations were incorrect.

Please read this advice about how to ask questions in the best way to
get answers: Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/3j77O3pOCRc/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/3j77O3pOCRc/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

17

I want it to be automatic . Is it possible?

On Thu, May 16, 2013 at 2:57 PM, Clinton Gormley clint@traveljury.comwrote:

Use delete-by-query

On 15 May 2013 20:17, Aakash Anuj aakashanuj.iitkgp@gmail.com wrote:

You were right I guess....becoz now my logs are getting deleted after the
specified time after the indexing, not the time stamp value. Any idea how
to automatically delete the logs relative to the time stamp value?

On Wednesday, May 15, 2013, Clinton Gormley wrote:

I've just asked, and apparently I'm incorrect. ttl and timestamp ARE
related.

But: i see a "@timestamp" field in your mapping, but no _timestamp path
mapping to point to that field
Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

On 15 May 2013 17:32, Aakash Anuj aakashanuj.iitkgp@gmail.com wrote:

Oh, I guess the last point was where I was fumbling.

I had read on a blog somewhere that the TTL applies relative to the
timestamp. So do you mean that the TTL applies relative to the system time
of creation?

On Wed, May 15, 2013 at 8:55 PM, Clinton Gormley clint@traveljury.comwrote:

On 15 May 2013 16:53, Aakash Anuj aakashanuj.iitkgp@gmail.com wrote:

But how to I drop the indices which are old automatically from the
ELASTIC SEARCH database?

you don't. you use a cron daemon to run a job once a day

And please explain me one more thing.....
When is the file defaul.json loaded, that is, when is the TTL
enabled? Is it when Elasticsearch runs?

Only when you restart a node. And its values only apply when you
create a new index

Also, how do I actually know if the indices get deleted?

You make your cron job send you an email

Even if I add logs which have a TTL of 1 day and a timestamp sometime
before 1 day, they get added to ES. Why ?

The ttl depends on the time you index it, not the timestamp field.

This could explain why things weren't working the way you expected.
Your expectations were incorrect.

Please read this advice about how to ask questions in the best way to
get answers: Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/3j77O3pOCRc/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/3j77O3pOCRc/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/3j77O3pOCRc/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

18

No

On 16 May 2013 11:28, Aakash Anuj aakashanuj.iitkgp@gmail.com wrote:

I want it to be automatic . Is it possible?

On Thu, May 16, 2013 at 2:57 PM, Clinton Gormley clint@traveljury.comwrote:

Use delete-by-query

On 15 May 2013 20:17, Aakash Anuj aakashanuj.iitkgp@gmail.com wrote:

You were right I guess....becoz now my logs are getting deleted after
the specified time after the indexing, not the time stamp value. Any idea
how to automatically delete the logs relative to the time stamp value?

On Wednesday, May 15, 2013, Clinton Gormley wrote:

I've just asked, and apparently I'm incorrect. ttl and timestamp ARE
related.

But: i see a "@timestamp" field in your mapping, but no _timestamp path
mapping to point to that field
Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

On 15 May 2013 17:32, Aakash Anuj aakashanuj.iitkgp@gmail.com wrote:

Oh, I guess the last point was where I was fumbling.

I had read on a blog somewhere that the TTL applies relative to the
timestamp. So do you mean that the TTL applies relative to the system time
of creation?

On Wed, May 15, 2013 at 8:55 PM, Clinton Gormley <clint@traveljury.com

wrote:

On 15 May 2013 16:53, Aakash Anuj aakashanuj.iitkgp@gmail.comwrote:

But how to I drop the indices which are old automatically from the
ELASTIC SEARCH database?

you don't. you use a cron daemon to run a job once a day

And please explain me one more thing.....
When is the file defaul.json loaded, that is, when is the TTL
enabled? Is it when Elasticsearch runs?

Only when you restart a node. And its values only apply when you
create a new index

Also, how do I actually know if the indices get deleted?

You make your cron job send you an email

Even if I add logs which have a TTL of 1 day and a timestamp
sometime before 1 day, they get added to ES. Why ?

The ttl depends on the time you index it, not the timestamp field.

This could explain why things weren't working the way you expected.
Your expectations were incorrect.

Please read this advice about how to ask questions in the best way to
get answers: Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/3j77O3pOCRc/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/3j77O3pOCRc/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/3j77O3pOCRc/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.