Sorting and Pagination

Elastic Stack Elasticsearch
1

I have logstash indicies that go back thirty days. I have logs in those
indices from today.

If I do a search with:

"size": 500,
"sort": [
{
"@timestamp": {
"order": "desc",
"ignore_unmapped": true
}
}
]

I don't get any logs from today. If I limit the search results to just
today by adding a time filter, I see the results from today:

"filter": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"from": 1410815562837,
"to": 1410901962837
}
}
}
]
}
}

Shouldn't ES be sorting on the entire result set and then return me the
top 500? Instead, it appears to limit the results to 500 and then sort by
time. Is this expected behavior? Is there a workaround?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8c5d33b9-1333-4324-891c-c382c8a41dde%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Another workaround appears to be by increasing the size to include all
matches. I'm using Kibana and by default the grid limits the search size
to 500 in pages of 100. By increasing the size to a much larger limit
which encompasses all matches, the sorting is right. I don't see any
performance hits that are obvious. Is this a known issue?

On Tuesday, September 16, 2014 5:15:11 PM UTC-4, Matt Hughes wrote:

I have logstash indicies that go back thirty days. I have logs in those
indices from today.

If I do a search with:

"size": 500,
"sort": [
{
"@timestamp": {
"order": "desc",
"ignore_unmapped": true
}
}
]

I don't get any logs from today. If I limit the search results to just
today by adding a time filter, I see the results from today:

"filter": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"from": 1410815562837,
"to": 1410901962837
}
}
}
]
}
}

Shouldn't ES be sorting on the entire result set and then return me the
top 500? Instead, it appears to limit the results to 500 and then sort by
time. Is this expected behavior? Is there a workaround?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/96eabb87-c5f1-4e27-b7d0-029b9acd75a5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.