Sort by _timestamp not working

Elastic Stack Elasticsearch
1

Hi,
I have a simple query and am trying to sort by "_timestamp", but it doesn't appear to work. Neither does sorting by the "_id" field. But, sorting works for other fields whose names don't begin with an underscore. Is there a restriction on fields starting with underscore?

My query URL is:
http://127.0.0.1:9200/_search?q=*&fields=_source,_timestamp&size=10&pretty&sort=_timestamp:asc

My mapping is:
{
"settings": {
"number_of_shards" : 5,
"number_of_replicas" : 1
},
"mappings" : {
"message" : {
"_source" : {"enabled" : true, "includes" : ["*"]},
"_all" : {"enabled": false},
"_timestamp" : {"enabled" : true, "index" : "analyzed", "store" : "yes"},
"properties" : {
"custID" : {"type" : "integer", "index" : "analyzed"},
"payload" : {"type" : "string", "index" : "not_analyzed"}
}
}
}
}

Thanks,
-Vinh

--

2

Never used the timestamp feature (I really should), but you should stick to
the default of non_analyzed instead of analyzed. Is the field indexing
correctly? Can you query on it? You should be able to. Since you are
analyzing the field, there is the possibility that the default analyzer is
splitting up the field into multiple terms. You cannot sort on a
multi-valued field. There would be an error in the log telling you so.

The id field is not indexed, therefore you cannot sort on it. You can
change to to be indexed:

Cheers,

Ivan

On Wed, Jan 16, 2013 at 3:26 PM, vinh vinh@loggly.com wrote:

enabled" : true, "index" : "analyzed",

--

3

Thanks Ivan,
My input value for "_timestamp" is a long, so I assume ES can't do much with it even if it attempted to analyze it. I tried changing the mapping to "not_analyzed" though, yet still no luck. I also updated the mapping for "_id" to be indexed, and still no luck trying to sort on that field, too.

To work around this, I defined a custom "timestamp" field in my mapping and am saving the long value to that field:
"properties" : {
"custID" : {"type" : "integer", "index" : "analyzed"},
"payload" : {"type" : "string", "index" : "not_analyzed"},
"timestamp": {"type" : "date", "index" : "not_analyzed"},
},

Now, I am finally able to sort by the new "timestamp" field. So it appears to me that sorting doesn't work any fields whose names are preceded by underscore. Just wasn't sure if this is expected ES behavior. If this is the case, I need to make a slight change in design and avoid storing values to ES's default/internal "_xxx" fields.

One result is now having 2 timestamp fields. But perhaps that's a good approach because at least for debugging, I can always refer to the original "_timestamp" field to know exactly when ES received the event, as opposed to my new "timestamp" field which gets its value from the application.

On Jan 16, 2013, at 3:40 PM, Ivan Brusic ivan@brusic.com wrote:

Never used the timestamp feature (I really should), but you should stick to the default of non_analyzed instead of analyzed. Is the field indexing correctly? Can you query on it? You should be able to. Since you are analyzing the field, there is the possibility that the default analyzer is splitting up the field into multiple terms. You cannot sort on a multi-valued field. There would be an error in the log telling you so.

The id field is not indexed, therefore you cannot sort on it. You can change to to be indexed: Elasticsearch Platform — Find real-time answers at scale | Elastic

Cheers,

Ivan

On Wed, Jan 16, 2013 at 3:26 PM, vinh vinh@loggly.com wrote:
enabled" : true, "index" : "analyzed",

--

--

4

Timestamp seems to work fine for me https://groups.google.com/forum/#!topic/elasticsearch/0AWUInB9-uY · GitHub

Are you sure your mapping is getting applied?

On Wednesday, January 16, 2013 7:58:16 PM UTC-5, vinh wrote:

Thanks Ivan,
My input value for "_timestamp" is a long, so I assume ES can't do much
with it even if it attempted to analyze it. I tried changing the mapping
to "not_analyzed" though, yet still no luck. I also updated the mapping
for "_id" to be indexed, and still no luck trying to sort on that field,
too.

To work around this, I defined a custom "timestamp" field in my mapping
and am saving the long value to that field:
"properties" : {
"custID" : {"type" : "integer", "index" : "analyzed"},
"payload" : {"type" : "string", "index" : "not_analyzed"},
"timestamp": {"type" : "date", "index" : "not_analyzed"},
},

Now, I am finally able to sort by the new "timestamp" field. So it
appears to me that sorting doesn't work any fields whose names are preceded
by underscore. Just wasn't sure if this is expected ES behavior. If this
is the case, I need to make a slight change in design and avoid storing
values to ES's default/internal "_xxx" fields.

One result is now having 2 timestamp fields. But perhaps that's a good
approach because at least for debugging, I can always refer to the original
"_timestamp" field to know exactly when ES received the event, as opposed
to my new "timestamp" field which gets its value from the application.

On Jan 16, 2013, at 3:40 PM, Ivan Brusic <iv...@brusic.com <javascript:>>
wrote:

Never used the timestamp feature (I really should), but you should stick
to the default of non_analyzed instead of analyzed. Is the field indexing
correctly? Can you query on it? You should be able to. Since you are
analyzing the field, there is the possibility that the default analyzer is
splitting up the field into multiple terms. You cannot sort on a
multi-valued field. There would be an error in the log telling you so.

The id field is not indexed, therefore you cannot sort on it. You can
change to to be indexed:
Elasticsearch Platform — Find real-time answers at scale | Elastic

Cheers,

Ivan

On Wed, Jan 16, 2013 at 3:26 PM, vinh <vi...@loggly.com <javascript:>>wrote:

enabled" : true, "index" : "analyzed",

--

--

5

Thanks Ivan,
Your test works. I even increased it to loop and insert >100 docs. I'll have to revisit my schema again, and to make sure it's applied correctly.
-Vinh

On Jan 16, 2013, at 7:02 PM, Igor Motov imotov@gmail.com wrote:

Timestamp seems to work fine for me https://groups.google.com/forum/#!topic/elasticsearch/0AWUInB9-uY · GitHub

Are you sure your mapping is getting applied?

On Wednesday, January 16, 2013 7:58:16 PM UTC-5, vinh wrote:
Thanks Ivan,
My input value for "_timestamp" is a long, so I assume ES can't do much with it even if it attempted to analyze it. I tried changing the mapping to "not_analyzed" though, yet still no luck. I also updated the mapping for "_id" to be indexed, and still no luck trying to sort on that field, too.

To work around this, I defined a custom "timestamp" field in my mapping and am saving the long value to that field:
"properties" : {
"custID" : {"type" : "integer", "index" : "analyzed"},
"payload" : {"type" : "string", "index" : "not_analyzed"},
"timestamp": {"type" : "date", "index" : "not_analyzed"},
},

Now, I am finally able to sort by the new "timestamp" field. So it appears to me that sorting doesn't work any fields whose names are preceded by underscore. Just wasn't sure if this is expected ES behavior. If this is the case, I need to make a slight change in design and avoid storing values to ES's default/internal "_xxx" fields.

One result is now having 2 timestamp fields. But perhaps that's a good approach because at least for debugging, I can always refer to the original "_timestamp" field to know exactly when ES received the event, as opposed to my new "timestamp" field which gets its value from the application.

On Jan 16, 2013, at 3:40 PM, Ivan Brusic iv...@brusic.com wrote:

Never used the timestamp feature (I really should), but you should stick to the default of non_analyzed instead of analyzed. Is the field indexing correctly? Can you query on it? You should be able to. Since you are analyzing the field, there is the possibility that the default analyzer is splitting up the field into multiple terms. You cannot sort on a multi-valued field. There would be an error in the log telling you so.

The id field is not indexed, therefore you cannot sort on it. You can change to to be indexed: Elasticsearch Platform — Find real-time answers at scale | Elastic

Cheers,

Ivan

On Wed, Jan 16, 2013 at 3:26 PM, vinh vi...@loggly.com wrote:
enabled" : true, "index" : "analyzed",

--

--

--