Retrieving all log events sorted by timestamp

[oskargranlund]

I'm trying to retrieve all log events from an index by using python elasticsearch interface.

Currently I use this line of code in order to execute the query:
scan(es, index="filebeat-2019.03.19", scroll="2m", query={"query": {"match": {"tags": "tag"}}})

It fetches data but the data is not sorted in chronological order according to the @timestamp field, is it possible to fetch the all available data sorted by timestamp?

[dadoonet (David Pilato)]

Yes. You can sort with https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html

I believe you need to pass the sort part in the Python query parameter?

[oskargranlund]

I'm still struggeling with fetching sorted results. Do you mean something like this?
query={ "sort": [{"source": {"timestamp": {"order": "asc"}}}], "query": { "match": {"tags": "tag"}} }
Does this sort the result within each scroll or does it sort the entire search?
I want to fetch about a million sorted log events.

[dadoonet (David Pilato)]

@oskargranlund

Do you mean something like this?

Yes.

@oskargranlund

Does this sort the result within each scroll or does it sort the entire search?

The entire search.

[oskargranlund]

Another question.. I am trying to use sort and query, however, they do not seem to work very well together. When having the sort line in the curl below everything is sorted correctly but the query is not applied to the results. When removing the sort line the result is not sorted but it is correctly filtered according to the query. What is going wrong? I've tried to move the position of the sort line to multiple positions with the same results.

res=$(curl -XGET --header 'Content-Type: application/json' host:port/index/_search?scroll=5m -d '{
"sort": [{"timestamp": {"order": "asc"}}],
"query": {
"bool" : {
"must" : [
{ "match" : {"tags" : "x, y"}},
{ "match" : {"host.name" : "hostname"}}
]
}
},
"size": "10"
}
}')

[dadoonet (David Pilato)]

That's another question. You should open a new discussion about it. And please provide a full recreation script as described in About the Elasticsearch category. It will help to better understand what you are doing. Please, try to keep the example as simple as possible.

A full reproduction script will help readers to understand, reproduce and if needed fix your problem. It will also most likely help to get a faster answer.