Sort by dynamic fields and use search_after

[hello]

Version: 5.2.2

I'm currently sorting by dynamic fields and a timestamp and want to utilize the search_after property. Unfortunately, for certain fields that are missing the dynamic field, they return "Infinity" or "-Infinity" in the sort property result set. When passing back those Infinity values in any position as a String how ES originally returned it, ES seems to not return any hits.

Example sort against docs that don't have the dynamic field.

Has anyone had to overcome this before?

[raymyers]

I believe I'm having a related issue using ES 5.4.0. I have a date field which comes back with sort value 9223372036854776000 when missing. Using that value in search_after gives the error: No matching token for number_type [BIG_INTEGER]

I haven't found a workaround yet. It seems like search_after currently can't be used when sorting by an optional date field.

Hopefully someone can help determine if there is a fix or documentation update in order.

[jimczi (Jimferenczi)]

I answered on the github issue directly:

Issue: Dynamic field sort and search_after

@raymyers your problem is different. 9223372036854776000 cannot be encoded as a long (the internal encoding for dates). It seems that your json library messed up ES result because a missing date would return either Long.MAX_VALUE or Long.MIN_VALUE. What library are you using to communicate with ES ?

[raymyers]

@jimczi, thanks for looking into this. I'm using elasticsearch-ruby 5.0.4. The underlying JSON library is multi_json 1.12.1. By turning on logging of the HTTP responses, I've confirmed that the Ruby library is changing the value. I can work around it by replacing values greater than 9223372036854775807 when the search_after array is assembled.

I'll file an issue with the client library. Thanks again!