Elastic search _timestamp path

Elastic Stack Elasticsearch
1

I am using Logstash to export the logs onto the elastic search database.
The problem is that I want to specify the TTL(time to live) with each index
as well, which is working completely fine. Here is the code in
/mappings/_default/default.json

{
"default" : {
"_ttl" : { "enabled" : true ,"default":"10s"}
}
}

But now the challenge is to make this TTL work relative to the timestamp of
the doc instead of the system time. I have come to know that I will need
the "_timestamp path" for it and tried various things. But I get a cannot
parse exception.

Here is what my new default.json looks like

{
"default" : {
"_ttl" : { "enabled" : true ,"default":"10s"}
"_timestamp": {"enabled":true, "path":"@timestamp"}
}
}

Now I guess I am doing something wrong with the path, which gives me the
error.

Here "@timestamp" is the timestamp that I parse from the logs and is of the
format 2013-05-3T05:19:16.776Z .

Even when I add the format field like

    "_timestamp": {"enabled":true,

"path":"@timestamp","format":"YYYY-MM:ddTHH:mm:ss.SSSZ"}

I get an exception.

What do I do? Any help would be appreciated.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

This is the error that I get in the latter case:

{"@source":"stdin://aakash-VPCEB26FG/","@tags":["INFO,gamereportin,coreSlave1,ERR_SYSTEM"],"@fields":{"ts":["2013/05/13-05:19:16.776"],"year":["2013"],"monthnum":["05"],"monthday":["13"],"hour":["05"],"minute":["19"],"second":["16"],"_second":["776"],"type1":["INFO"],"slave":["coreSlave1"],"type2":["gamereportin"],"message":["[0000000000000000/00000000000000000000]
[GameReportingSlaveImpl:0x30bf7699a010].processReport() : Error processing
game report for id=18014398509852207, type=frostbite_multiplayer,
error=ERR_SYSTEM"]},"@timestamp":"2013-05-13T05:19:16.776Z","@source_host":"aakash-VPCEB26FG","@source_path":"/","@message":"[0000000000000000/00000000000000000000]
[GameReportingSlaveImpl:0x30bf7699a010].processReport() : Error processing
game report for id=18014398509852207, type=frostbite_multiplayer,
error=ERR_SYSTEM","@type":"stdin-type"}

Failed to index an event, will retry
{:exception=>org.elasticsearch.transport.RemoteTransportException: [Martha
Johansson][inet[/192.168.8.3:9300]][indices/create],
:event=>{"@source"=>"stdin://aakash-VPCEB26FG/",
"@tags"=>["INFO,gamereportin,coreSlave1,ERR_SYSTEM"],
"@fields"=>{"ts"=>["2013/05/13-05:19:16.776"], "year"=>["2013"],
"monthnum"=>["05"], "monthday"=>["13"], "hour"=>["05"], "minute"=>["19"],
"second"=>["16"], "_second"=>["776"], "type1"=>["INFO"],
"slave"=>["coreSlave1"], "type2"=>["gamereportin"],
"message"=>["[0000000000000000/00000000000000000000]
[GameReportingSlaveImpl:0x30bf7699a010].processReport() : Error processing
game report for id=18014398509852207, type=frostbite_multiplayer,
error=ERR_SYSTEM"]}, "@timestamp"=>"2013-05-13T05:19:16.776Z",
"@source_host"=>"aakash-VPCEB26FG", "@source_path"=>"/",
"@message"=>"[0000000000000000/00000000000000000000]
[GameReportingSlaveImpl:0x30bf7699a010].processReport() : Error processing
game report for id=18014398509852207, type=frostbite_multiplayer,
error=ERR_SYSTEM", "@type"=>"stdin-type"}, :level=>:warn}

On Fri, May 17, 2013 at 9:53 AM, Aakash Anuj aakashanuj.iitkgp@gmail.comwrote:

I am using Logstash to export the logs onto the Elasticsearch database.
The problem is that I want to specify the TTL(time to live) with each index
as well, which is working completely fine. Here is the code in
/mappings/_default/default.json

{
"default" : {
"_ttl" : { "enabled" : true ,"default":"10s"}
}
}

But now the challenge is to make this TTL work relative to the timestamp
of the doc instead of the system time. I have come to know that I will need
the "_timestamp path" for it and tried various things. But I get a cannot
parse exception.

Here is what my new default.json looks like

{
"default" : {
"_ttl" : { "enabled" : true ,"default":"10s"}
"_timestamp": {"enabled":true, "path":"@timestamp"}
}
}

Now I guess I am doing something wrong with the path, which gives me the
error.

Here "@timestamp" is the timestamp that I parse from the logs and is of
the format 2013-05-3T05:19:16.776Z .

Even when I add the format field like

    "_timestamp": {"enabled":true,

"path":"@timestamp","format":"YYYY-MM:ddTHH:mm:ss.SSSZ"}

I get an exception.

What do I do? Any help would be appreciated.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/ubAw9b1HCzE/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Hi Aakash

Well asked question! You've provided all the necessary info.

OK, you have a couple of problems here:

  1. Your format is incorrect:
    a) the T is a literal, and so needs to be written as "...dd'T'HH..."
    b) You have ':dd' but your timestamp uses '-dd'

  2. You are trying to set docs to expire at a time in the past. ES will
    just refuse to index those docs (which makes sense)

Also, the errors that you pasted are from logstash, not from ES. If you
look at the logs in ES itself, you'll get more of an idea of what the
problem is. Also, trying it out directly in ES will probably be more
revealing than doing it via logstash.

clint

On 17 May 2013 06:48, Aakash Anuj aakashanuj.iitkgp@gmail.com wrote:

This is the error that I get in the latter case:

{"@source":"stdin://aakash-VPCEB26FG/","@tags":["INFO,gamereportin,coreSlave1,ERR_SYSTEM"],"@fields":{"ts":["2013/05/13-05:19:16.776"],"year":["2013"],"monthnum":["05"],"monthday":["13"],"hour":["05"],"minute":["19"],"second":["16"],"_second":["776"],"type1":["INFO"],"slave":["coreSlave1"],"type2":["gamereportin"],"message":["[0000000000000000/00000000000000000000]
[GameReportingSlaveImpl:0x30bf7699a010].processReport() : Error processing
game report for id=18014398509852207, type=frostbite_multiplayer,
error=ERR_SYSTEM"]},"@timestamp":"2013-05-13T05:19:16.776Z","@source_host":"aakash-VPCEB26FG","@source_path":"/","@message":"[0000000000000000/00000000000000000000]
[GameReportingSlaveImpl:0x30bf7699a010].processReport() : Error processing
game report for id=18014398509852207, type=frostbite_multiplayer,
error=ERR_SYSTEM","@type":"stdin-type"}

Failed to index an event, will retry
{:exception=>org.elasticsearch.transport.RemoteTransportException: [Martha
Johansson][inet[/192.168.8.3:9300]][indices/create],
:event=>{"@source"=>"stdin://aakash-VPCEB26FG/",
"@tags"=>["INFO,gamereportin,coreSlave1,ERR_SYSTEM"],
"@fields"=>{"ts"=>["2013/05/13-05:19:16.776"], "year"=>["2013"],
"monthnum"=>["05"], "monthday"=>["13"], "hour"=>["05"], "minute"=>["19"],
"second"=>["16"], "_second"=>["776"], "type1"=>["INFO"],
"slave"=>["coreSlave1"], "type2"=>["gamereportin"],
"message"=>["[0000000000000000/00000000000000000000]
[GameReportingSlaveImpl:0x30bf7699a010].processReport() : Error processing
game report for id=18014398509852207, type=frostbite_multiplayer,
error=ERR_SYSTEM"]}, "@timestamp"=>"2013-05-13T05:19:16.776Z",
"@source_host"=>"aakash-VPCEB26FG", "@source_path"=>"/",
"@message"=>"[0000000000000000/00000000000000000000]
[GameReportingSlaveImpl:0x30bf7699a010].processReport() : Error processing
game report for id=18014398509852207, type=frostbite_multiplayer,
error=ERR_SYSTEM", "@type"=>"stdin-type"}, :level=>:warn}

On Fri, May 17, 2013 at 9:53 AM, Aakash Anuj aakashanuj.iitkgp@gmail.comwrote:

I am using Logstash to export the logs onto the Elasticsearch database.
The problem is that I want to specify the TTL(time to live) with each index
as well, which is working completely fine. Here is the code in
/mappings/_default/default.json

{
"default" : {
"_ttl" : { "enabled" : true ,"default":"10s"}
}
}

But now the challenge is to make this TTL work relative to the timestamp
of the doc instead of the system time. I have come to know that I will need
the "_timestamp path" for it and tried various things. But I get a cannot
parse exception.

Here is what my new default.json looks like

{
"default" : {
"_ttl" : { "enabled" : true ,"default":"10s"}
"_timestamp": {"enabled":true, "path":"@timestamp"}
}
}

Now I guess I am doing something wrong with the path, which gives me the
error.

Here "@timestamp" is the timestamp that I parse from the logs and is of
the format 2013-05-3T05:19:16.776Z .

Even when I add the format field like

    "_timestamp": {"enabled":true,

"path":"@timestamp","format":"YYYY-MM:ddTHH:mm:ss.SSSZ"}

I get an exception.

What do I do? Any help would be appreciated.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/ubAw9b1HCzE/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Aakash Anuj,
Junior Undergraduate,
Department of Computer Science and Engineering,
Indian Institute of Technology, Kharagpur.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.