_timestamp

vinh · March 5, 2013, 10:29pm

When inserting docs, I store my own timestamp value in the default "_timestamp" field. Then I search docs and fetch both "_source" and "_timestamp" fields. I notice that the value of _source._timestamp correctly has my value. But, fields._timestamp is different.

Is ES maintaining _timestamp in two different places, even if I am overriding it?

}, {
  "_index" : "myindex",
  "_type" : "message",
  "_id" : "3c60f48b-7e28-11e2-8081-000c29c275f9",
  "_score" : null, "_source" : {"_id":"3c60f48b-7e28-11e2-8081-000c29c275f9","timestamp":1361672205131,"msg":"hello there"},
  "fields" : {
    "_timestamp" : 1361672211995
  },
  "sort" : [ 1361672211995 ]
} ]

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

vinh · March 5, 2013, 10:41pm

Sorry, copy-paste error. The ES response is:

}, {
  "_index" : "myindex",
  "_type" : "message",
  "_id" : "3c60f48b-7e28-11e2-8081-000c29c275f9",
  "_score" : null, "_source" : {"_id":"3c60f48b-7e28-11e2-8081-000c29c275f9","_timestamp":1361672205131,"msg":"hello there"},
  "fields" : {
    "_timestamp" : 1361672211995
  },
  "sort" : [ 1361672211995 ]
} ]

On Mar 5, 2013, at 2:29 PM, vinh vinh@loggly.com wrote:

When inserting docs, I store my own timestamp value in the default "_timestamp" field. Then I search docs and fetch both "_source" and "_timestamp" fields. I notice that the value of _source._timestamp correctly has my value. But, fields._timestamp is different.

Is ES maintaining _timestamp in two different places, even if I am overriding it?
}, {
  "_index" : "myindex",
  "_type" : "message",
  "_id" : "3c60f48b-7e28-11e2-8081-000c29c275f9",
  "_score" : null, "_source" : {"_id":"3c60f48b-7e28-11e2-8081-000c29c275f9","timestamp":1361672205131,"msg":"hello there"},
  "fields" : {
    "_timestamp" : 1361672211995
  },
  "sort" : [ 1361672211995 ]
} ]

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Ivan · March 5, 2013, 10:41pm

Did you enable the timestamp field in your mapping? It is not enabled by
default. Also, your example shows the field as "timestamp" without the
underscore. Is this correct? Your source should contain whatever was
inserted.

--
Ivan

On Tue, Mar 5, 2013 at 2:29 PM, vinh vinh@loggly.com wrote:

When inserting docs, I store my own timestamp value in the default
"_timestamp" field. Then I search docs and fetch both "_source" and
"_timestamp" fields. I notice that the value of _source._timestamp
correctly has my value. But, fields._timestamp is different.

Is ES maintaining _timestamp in two different places, even if I am
overriding it?
}, {
  "_index" : "myindex",
  "_type" : "message",
  "_id" : "3c60f48b-7e28-11e2-8081-000c29c275f9",
  "_score" : null, "_source" :
{"_id":"3c60f48b-7e28-11e2-8081-000c29c275f9","timestamp":1361672205131,"msg":"hello
there"},
"fields" : {
"_timestamp" : 1361672211995
},
"sort" : [ 1361672211995 ]
} ]

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

vinh · March 5, 2013, 10:43pm

My schema is:

{
"template" : "*",
"settings": {
"number_of_shards" : 5,
"number_of_replicas" : 1
},
"mappings" : {
"message" : {
"_source" : {"enabled" : true},
"_all" : {"enabled": false},
"_id" : {"index": "not_analyzed", "store" : "yes"},
"_timestamp" : {"enabled" : true, "index" : "not_analyzed", "store" : "yes"}
}
}
}

On Mar 5, 2013, at 2:41 PM, vinh vinh@loggly.com wrote:

Sorry, copy-paste error. The ES response is:
}, {
  "_index" : "myindex",
  "_type" : "message",
  "_id" : "3c60f48b-7e28-11e2-8081-000c29c275f9",
  "_score" : null, "_source" : {"_id":"3c60f48b-7e28-11e2-8081-000c29c275f9","_timestamp":1361672205131,"msg":"hello there"},
  "fields" : {
    "_timestamp" : 1361672211995
  },
  "sort" : [ 1361672211995 ]
} ]
On Mar 5, 2013, at 2:29 PM, vinh vinh@loggly.com wrote:
When inserting docs, I store my own timestamp value in the default "_timestamp" field. Then I search docs and fetch both "_source" and "_timestamp" fields. I notice that the value of _source._timestamp correctly has my value. But, fields._timestamp is different.

Is ES maintaining _timestamp in two different places, even if I am overriding it?
}, {
  "_index" : "myindex",
  "_type" : "message",
  "_id" : "3c60f48b-7e28-11e2-8081-000c29c275f9",
  "_score" : null, "_source" : {"_id":"3c60f48b-7e28-11e2-8081-000c29c275f9","timestamp":1361672205131,"msg":"hello there"},
  "fields" : {
    "_timestamp" : 1361672211995
  },
  "sort" : [ 1361672211995 ]
} ]

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · March 5, 2013, 10:47pm

Have you tried without "index" : "not_analyzed", "store" : "yes" ?

Jörg

Am 05.03.13 23:43, schrieb vinh:

"_timestamp" : {"enabled" : true, "index" : "not_analyzed",

"store" : "yes"}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Ivan · March 5, 2013, 10:52pm

I have never used the timestamp field, but I have seen peculiarities with
the boost field.

From what I can tell from the documentation, the timestamp field is added
automatically. Providing your own is conflicting with the internal version.
And while I do not know all the details of the conflict resolution
mechanism, my only advice would be to simply not get into the situation in
the first place. Either you add the timestamp or let Elasticsearch do it.
As you can see, maintaining your own is not difficult.

It appears that Elasticsearch is preserving the original in the source, but
adding its own as the indexed field. The behavior seems ideal.

Cheers,
Ivan

On Tue, Mar 5, 2013 at 2:47 PM, Jörg Prante joergprante@gmail.com wrote:

Have you tried without "index" : "not_analyzed", "store" : "yes" ?

Jörg

Am 05.03.13 23:43, schrieb vinh:
 "_timestamp" : {"enabled" : true, "index" : "not_analyzed", "store" :
"yes"}

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@**googlegroups.com elasticsearch%2Bunsubscribe@googlegroups.com
.
For more options, visit https://groups.google.com/**groups/opt_out https://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Benjamin_Deveze · March 5, 2013, 11:23pm

Hey you can either set the timestamp value in your index request or extract
it from the source. If you want to extract it from the source you have to
tell ES where to find you timestamp field using path in your mapping as
described here
Elasticsearch Platform — Find real-time answers at scale | Elastic.

It won't automatically use a _timestamp field set in your source content.

On Tuesday, March 5, 2013 11:29:44 PM UTC+1, vinh wrote:

When inserting docs, I store my own timestamp value in the default
"_timestamp" field. Then I search docs and fetch both "_source" and
"_timestamp" fields. I notice that the value of _source._timestamp
correctly has my value. But, fields._timestamp is different.

Is ES maintaining _timestamp in two different places, even if I am
overriding it?
}, {
  "_index" : "myindex",
  "_type" : "message",
  "_id" : "3c60f48b-7e28-11e2-8081-000c29c275f9",
  "_score" : null, "_source" : 
{"_id":"3c60f48b-7e28-11e2-8081-000c29c275f9","timestamp":1361672205131,"msg":"hello
there"},
"fields" : {
"_timestamp" : 1361672211995
},
"sort" : [ 1361672211995 ]
} ]

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

vinh · March 6, 2013, 6:34am

Thanks Jorg,
No luck without that setting. But I also need that setting, or else I can't sort on _timestamp.
-Vinh

On Mar 5, 2013, at 2:47 PM, Jörg Prante joergprante@gmail.com wrote:

Have you tried without "index" : "not_analyzed", "store" : "yes" ?

Jörg

Am 05.03.13 23:43, schrieb vinh:

"_timestamp" : {"enabled" : true, "index" : "not_analyzed", "store" : "yes"}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

vinh · March 6, 2013, 6:48am

Thanks Benjamin,

I am currently setting the _timestamp value in the request. I'm trying to avoid mapping/extracting it to another field in the source, to avoid needing two fields for the same value.

-Vinh

On Mar 5, 2013, at 3:23 PM, Benjamin Devèze benjamin.deveze@gmail.com wrote:

Hey you can either set the timestamp value in your index request or extract it from the source. If you want to extract it from the source you have to tell ES where to find you timestamp field using path in your mapping as described here
Elasticsearch Platform — Find real-time answers at scale | Elastic.

It won't automatically use a _timestamp field set in your source content.

On Tuesday, March 5, 2013 11:29:44 PM UTC+1, vinh wrote:
When inserting docs, I store my own timestamp value in the default "_timestamp" field. Then I search docs and fetch both "_source" and "_timestamp" fields. I notice that the value of _source._timestamp correctly has my value. But, fields._timestamp is different.

Is ES maintaining _timestamp in two different places, even if I am overriding it?
}, {
  "_index" : "myindex",
  "_type" : "message",
  "_id" : "3c60f48b-7e28-11e2-8081-000c29c275f9",
  "_score" : null, "_source" : {"_id":"3c60f48b-7e28-11e2-8081-000c29c275f9","timestamp":1361672205131,"msg":"hello there"},
  "fields" : {
    "_timestamp" : 1361672211995
  },
  "sort" : [ 1361672211995 ]
} ]
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.