# Elastic search \_timestamp path **URL:** \ **Category:** Elasticsearch\ **Created:** [May 17, 2013, 4:23am UTC](https://discuss.elastic.co/t/elastic-search--timestamp-path/12004 "2013-05-17T04:23:23Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![aakashanuj](https://avatars.discourse-cdn.com/v4/letter/a/c68b51/32.png) [@aakashanuj](https://discuss.elastic.co/u/aakashanuj)\ **Post date:** [May 17, 2013, 4:23am UTC](https://discuss.elastic.co/t/elastic-search--timestamp-path/12004/1 "2013-05-17T04:23:23Z")
I am using Logstash to export the logs onto the elastic search database. The problem is that I want to specify the TTL(time to live) with each index as well, which is working completely fine. Here is the code in /mappings/\_default/_default_.json { "_default_" : { "\_ttl" : { "enabled" : true ,"default":"10s"} } } But now the challenge is to make this TTL work relative to the timestamp of the doc instead of the system time. I have come to know that I will need the "\_timestamp path" for it and tried various things. But I get a cannot parse exception. Here is what my new _default_.json looks like { "_default_" : { "\_ttl" : { "enabled" : true ,"default":"10s"} "\_timestamp": {"enabled":true, "path":"@timestamp"} } } Now I guess I am doing something wrong with the path, which gives me the error. Here "@timestamp" is the timestamp that I parse from the logs and is of the format 2013-05-3T05:19:16.776Z . Even when I add the format field like ``` "_timestamp": {"enabled":true, ``` "path":"@timestamp","format":"YYYY-MM:ddTHH:mm:ss.SSSZ"} I get an exception. What do I do? Any help would be appreciated. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![aakashanuj](https://avatars.discourse-cdn.com/v4/letter/a/c68b51/32.png) [@aakashanuj](https://discuss.elastic.co/u/aakashanuj)\ **Post date:** [May 17, 2013, 4:48am UTC](https://discuss.elastic.co/t/elastic-search--timestamp-path/12004/2 "2013-05-17T04:48:20Z")
This is the error that I get in the latter case: {"@source":"stdin://aakash-VPCEB26FG/","@tags":["INFO,gamereportin,coreSlave1,ERR\_SYSTEM"],"@fields":{"ts":["2013/05/13-05:19:16.776"],"year":["2013"],"monthnum":["05"],"monthday":["13"],"hour":["05"],"minute":["19"],"second":["16"],"\_second":["776"],"type1":["INFO"],"slave":["coreSlave1"],"type2":["gamereportin"],"message":["[0000000000000000/00000000000000000000] [GameReportingSlaveImpl:0x30bf7699a010].processReport() : Error processing game report for id=18014398509852207, type=frostbite\_multiplayer, error=ERR\_SYSTEM"]},"@timestamp":"2013-05-13T05:19:16.776Z","@source\_host":"aakash-VPCEB26FG","@source\_path":"/","@message":"[0000000000000000/00000000000000000000] [GameReportingSlaveImpl:0x30bf7699a010].processReport() : Error processing game report for id=18014398509852207, type=frostbite\_multiplayer, error=ERR\_SYSTEM","@type":"stdin-type"} Failed to index an event, will retry {:exception=\>org.elasticsearch.transport.RemoteTransportException: [Martha Johansson][inet[/192.168.8.3:9300]][indices/create], :event=\>{"@source"=\>"stdin://aakash-VPCEB26FG/", "@tags"=\>["INFO,gamereportin,coreSlave1,ERR\_SYSTEM"], "@fields"=\>{"ts"=\>["2013/05/13-05:19:16.776"], "year"=\>["2013"], "monthnum"=\>["05"], "monthday"=\>["13"], "hour"=\>["05"], "minute"=\>["19"], "second"=\>["16"], "\_second"=\>["776"], "type1"=\>["INFO"], "slave"=\>["coreSlave1"], "type2"=\>["gamereportin"], "message"=\>["[0000000000000000/00000000000000000000] [GameReportingSlaveImpl:0x30bf7699a010].processReport() : Error processing game report for id=18014398509852207, type=frostbite\_multiplayer, error=ERR\_SYSTEM"]}, "@timestamp"=\>"2013-05-13T05:19:16.776Z", "@source\_host"=\>"aakash-VPCEB26FG", "@source\_path"=\>"/", "@message"=\>"[0000000000000000/00000000000000000000] [GameReportingSlaveImpl:0x30bf7699a010].processReport() : Error processing game report for id=18014398509852207, type=frostbite\_multiplayer, error=ERR\_SYSTEM", "@type"=\>"stdin-type"}, :level=\>:warn} On Fri, May 17, 2013 at 9:53 AM, Aakash Anuj [aakashanuj.iitkgp@gmail.com](mailto:aakashanuj.iitkgp@gmail.com)wrote: > I am using Logstash to export the logs onto the Elasticsearch database. > The problem is that I want to specify the TTL(time to live) with each index > as well, which is working completely fine. Here is the code in > /mappings/\_default/_default_.json > > { > "_default_" : { > "\_ttl" : { "enabled" : true ,"default":"10s"} > } > } > > But now the challenge is to make this TTL work relative to the timestamp > of the doc instead of the system time. I have come to know that I will need > the "\_timestamp path" for it and tried various things. But I get a cannot > parse exception. > > Here is what my new _default_.json looks like > > { > "_default_" : { > "\_ttl" : { "enabled" : true ,"default":"10s"} > "\_timestamp": {"enabled":true, "path":"@timestamp"} > } > } > > Now I guess I am doing something wrong with the path, which gives me the > error. > > Here "@timestamp" is the timestamp that I parse from the logs and is of > the format 2013-05-3T05:19:16.776Z . > > Even when I add the format field like > > ``` > "_timestamp": {"enabled":true, > > ``` > > "path":"@timestamp","format":"YYYY-MM:ddTHH:mm:ss.SSSZ"} > > I get an exception. > > What do I do? Any help would be appreciated. > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/ubAw9b1HCzE/unsubscribe?hl=en-US](https://groups.google.com/d/topic/elasticsearch/ubAw9b1HCzE/unsubscribe?hl=en-US) > . > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Regards, Aakash Anuj, Junior Undergraduate, Department of Computer Science and Engineering, Indian Institute of Technology, Kharagpur. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [May 17, 2013, 11:14am UTC](https://discuss.elastic.co/t/elastic-search--timestamp-path/12004/3 "2013-05-17T11:14:34Z")
Hi Aakash Well asked question! You've provided all the necessary info. OK, you have a couple of problems here: 1. Your format is incorrect: a) the T is a literal, and so needs to be written as "...dd'T'HH..." b) You have ':dd' but your timestamp uses '-dd' 2. You are trying to set docs to expire at a time in the past. ES will just refuse to index those docs (which makes sense) Also, the errors that you pasted are from logstash, not from ES. If you look at the logs in ES itself, you'll get more of an idea of what the problem is. Also, trying it out directly in ES will probably be more revealing than doing it via logstash. clint On 17 May 2013 06:48, Aakash Anuj [aakashanuj.iitkgp@gmail.com](mailto:aakashanuj.iitkgp@gmail.com) wrote: > This is the error that I get in the latter case: > > {"@source":"stdin://aakash-VPCEB26FG/","@tags":["INFO,gamereportin,coreSlave1,ERR\_SYSTEM"],"@fields":{"ts":["2013/05/13-05:19:16.776"],"year":["2013"],"monthnum":["05"],"monthday":["13"],"hour":["05"],"minute":["19"],"second":["16"],"\_second":["776"],"type1":["INFO"],"slave":["coreSlave1"],"type2":["gamereportin"],"message":["[0000000000000000/00000000000000000000] > [GameReportingSlaveImpl:0x30bf7699a010].processReport() : Error processing > game report for id=18014398509852207, type=frostbite\_multiplayer, > error=ERR\_SYSTEM"]},"@timestamp":"2013-05-13T05:19:16.776Z","@source\_host":"aakash-VPCEB26FG","@source\_path":"/","@message":"[0000000000000000/00000000000000000000] > [GameReportingSlaveImpl:0x30bf7699a010].processReport() : Error processing > game report for id=18014398509852207, type=frostbite\_multiplayer, > error=ERR\_SYSTEM","@type":"stdin-type"} > > Failed to index an event, will retry > {:exception=\>org.elasticsearch.transport.RemoteTransportException: [Martha > Johansson][inet[/192.168.8.3:9300]][indices/create], > :event=\>{"@source"=\>"stdin://aakash-VPCEB26FG/", > "@tags"=\>["INFO,gamereportin,coreSlave1,ERR\_SYSTEM"], > "@fields"=\>{"ts"=\>["2013/05/13-05:19:16.776"], "year"=\>["2013"], > "monthnum"=\>["05"], "monthday"=\>["13"], "hour"=\>["05"], "minute"=\>["19"], > "second"=\>["16"], "\_second"=\>["776"], "type1"=\>["INFO"], > "slave"=\>["coreSlave1"], "type2"=\>["gamereportin"], > "message"=\>["[0000000000000000/00000000000000000000] > [GameReportingSlaveImpl:0x30bf7699a010].processReport() : Error processing > game report for id=18014398509852207, type=frostbite\_multiplayer, > error=ERR\_SYSTEM"]}, "@timestamp"=\>"2013-05-13T05:19:16.776Z", > "@source\_host"=\>"aakash-VPCEB26FG", "@source\_path"=\>"/", > "@message"=\>"[0000000000000000/00000000000000000000] > [GameReportingSlaveImpl:0x30bf7699a010].processReport() : Error processing > game report for id=18014398509852207, type=frostbite\_multiplayer, > error=ERR\_SYSTEM", "@type"=\>"stdin-type"}, :level=\>:warn} > > On Fri, May 17, 2013 at 9:53 AM, Aakash Anuj [aakashanuj.iitkgp@gmail.com](mailto:aakashanuj.iitkgp@gmail.com)wrote: > > > I am using Logstash to export the logs onto the Elasticsearch database. > > The problem is that I want to specify the TTL(time to live) with each index > > as well, which is working completely fine. Here is the code in > > /mappings/\_default/_default_.json > > > > { > > "_default_" : { > > "\_ttl" : { "enabled" : true ,"default":"10s"} > > } > > } > > > > But now the challenge is to make this TTL work relative to the timestamp > > of the doc instead of the system time. I have come to know that I will need > > the "\_timestamp path" for it and tried various things. But I get a cannot > > parse exception. > > > > Here is what my new _default_.json looks like > > > > { > > "_default_" : { > > "\_ttl" : { "enabled" : true ,"default":"10s"} > > "\_timestamp": {"enabled":true, "path":"@timestamp"} > > } > > } > > > > Now I guess I am doing something wrong with the path, which gives me the > > error. > > > > Here "@timestamp" is the timestamp that I parse from the logs and is of > > the format 2013-05-3T05:19:16.776Z . > > > > Even when I add the format field like > > > > ``` > > "_timestamp": {"enabled":true, > > > > ``` > > > > "path":"@timestamp","format":"YYYY-MM:ddTHH:mm:ss.SSSZ"} > > > > I get an exception. > > > > What do I do? Any help would be appreciated. > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/ubAw9b1HCzE/unsubscribe?hl=en-US](https://groups.google.com/d/topic/elasticsearch/ubAw9b1HCzE/unsubscribe?hl=en-US) > > . > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > Regards, > Aakash Anuj, > Junior Undergraduate, > Department of Computer Science and Engineering, > Indian Institute of Technology, Kharagpur. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:36am UTC](https://discuss.elastic.co/t/elastic-search--timestamp-path/12004/4 "2017-07-06T02:36:07Z")