Sorting response hits using java api

Elastic Stack Elasticsearch
1

Hello,

i'm using logstash to manage logs, and as part of it - elasticsearch.
when querying the logs i'm using* scan search* and have the following
fields for each hit: @message, @source, @source_host, @source_path, @tags, *@timestamp,
*@type.

this is my problem - the returned hits are unsorted. I want to sort them by
@timestamp. How can I do so in java?
This is what I tried:

SearchResponse scrollResp1 = client.prepareSearch()
.setSearchType(SearchType.SCAN)
.setScroll(new TimeValue(600000))
.setQuery(QueryBuilders.queryString("@tags:"" +
tagToSearch + "" AND @tags:"" + buildNumber + "" AND @source_path:"" +
fileName + """))
.setSize(100).addSort("@timestamp", SortOrder.ASC)
.execute().actionGet();

and then the scrolling for scan search:

        while (true) {
            scrollResp1 =

client.prepareSearchScroll(scrollResp1.getScrollId()).setScroll(new
TimeValue(600000)).execute().actionGet();
String message;

            for (SearchHit hit : scrollResp1.getHits()) {
                     ..................
            }

but it didn't sort anything..

any suggestions?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

You cannot sort scan searches.

--
Ivan

On Mon, Aug 12, 2013 at 8:28 AM, Nir Biran nbiran2@gmail.com wrote:

Hello,

i'm using logstash to manage logs, and as part of it - elasticsearch.
when querying the logs i'm using* scan search* and have the following
fields for each hit: @message, @source, @source_host, @source_path, @tags,
*@timestamp, *@type.

this is my problem - the returned hits are unsorted. I want to sort them
by @timestamp. How can I do so in java?
This is what I tried:

SearchResponse scrollResp1 = client.prepareSearch()
.setSearchType(SearchType.SCAN)
.setScroll(new TimeValue(600000))
.setQuery(QueryBuilders.queryString("@tags:"" +
tagToSearch + "" AND @tags:"" + buildNumber + "" AND @source_path:"" +
fileName + """))
.setSize(100).addSort("@timestamp", SortOrder.ASC)
.execute().actionGet();

and then the scrolling for scan search:

        while (true) {
            scrollResp1 =

client.prepareSearchScroll(scrollResp1.getScrollId()).setScroll(new
TimeValue(600000)).execute().actionGet();
String message;

            for (SearchHit hit : scrollResp1.getHits()) {
                     ..................
            }

but it didn't sort anything..

any suggestions?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Ivan,

Slight correction: Elasticsearch cannot sort scan searches. But I can!

My sorting of response documents is external to Elasticsearch, bolted on
after the query has returned. I use a TreeSet which keeps its members
sorted. I then implement a limit. When the limit is reached:

  1. If the document is outside the upper bound it is ignored.
  2. Otherwise, the document is added and the last document (at the upper
    bound) is removed (preserving the specified limit).

As I issue a scan query, I can then push documents into this TreeSet-based
sorter and always end up with the N-most sorted documents. This isn't
something I use a lot, but when I do need it this method is very, very
useful. And the additional time related to sorting (including creating
keys) is minimal.

Of course, there's lots of code to create the sort keys (probably
duplicates a lot of what ES is doing, if I could see its internal comments
:wink:

But I do have a side question: Is there a way to query (via Java, of
course) the types of each explicitly mapped field? I can create mappings,
but querying the mappings for this kind of thing eludes me. For now, I've
created my own "schema" definition, and then use it to (1) generate the ES
mappings with all the niggly add-ons (such as Finnish character
equivalencies), and then also use it to drive the collation key generation
for this post-query sorting. But if I could ask Elasticsearch directly at
run-time, then this process would become much simpler and more robust.

Brian

On Monday, August 12, 2013 11:55:48 AM UTC-4, Ivan Brusic wrote:

You cannot sort scan searches.

--
Ivan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Thanks for the replies.
Brian, I don't have an answer for you, but I have a follow up question to
my original post in light of the replies:

So I can't sort a scan search, but is there a different search which is
sortable AND returns more than the first 10 hits?
I only used scan search because it returns all the hits and not only the
first 10..

seems like a trivial request - return all hits sorted.
is there a trivial way?

On Tuesday, 13 August 2013 00:01:02 UTC+3, InquiringMind wrote:

Ivan,

Slight correction: Elasticsearch cannot sort scan searches. But I can!

My sorting of response documents is external to Elasticsearch, bolted on
after the query has returned. I use a TreeSet which keeps its members
sorted. I then implement a limit. When the limit is reached:

  1. If the document is outside the upper bound it is ignored.
  2. Otherwise, the document is added and the last document (at the upper
    bound) is removed (preserving the specified limit).

As I issue a scan query, I can then push documents into this TreeSet-based
sorter and always end up with the N-most sorted documents. This isn't
something I use a lot, but when I do need it this method is very, very
useful. And the additional time related to sorting (including creating
keys) is minimal.

Of course, there's lots of code to create the sort keys (probably
duplicates a lot of what ES is doing, if I could see its internal comments
:wink:

But I do have a side question: Is there a way to query (via Java, of
course) the types of each explicitly mapped field? I can create mappings,
but querying the mappings for this kind of thing eludes me. For now, I've
created my own "schema" definition, and then use it to (1) generate the ES
mappings with all the niggly add-ons (such as Finnish character
equivalencies), and then also use it to drive the collation key generation
for this post-query sorting. But if I could ask Elasticsearch directly at
run-time, then this process would become much simpler and more robust.

Brian

On Monday, August 12, 2013 11:55:48 AM UTC-4, Ivan Brusic wrote:

You cannot sort scan searches.

--
Ivan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

There is from/size

http://www.elasticsearch.org/guide/reference/api/search/from-size/

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.