I have an index in ES 6.6 used to search regular content that should be scored differently based on the user. Today we have more than 50 fields that can be used to calculate this score.
In order to speed up the query, I decided to use early termination like described here
I created a field based on a combination of all individual fields used for scoring and sorted the index with this value.
However, I can't sort the results of the query based on this single field (since the sort order will depend on each user), so I started using the
terminate_after parameter in the query.
After doing some evaluations I noticed that terminating the query after 1000 elements was good enough to have a set of 10 results with the optimal score for each user.
This seemed to work well and I got the speed up that I wanted, however later I realized that this only works if each shard in the index has only one segment. I call force_merge with value 1 during index creation, but I can't do that for index updates during normal traffic.
After updating the index, new segments are created and they are completely ignored by the query if we reach enough matches for the
terminate_after parameter in the original segment.
This happen even if in the new segment, the field that defines the index sorting would position the document as the top result.
My questions now:
Is that a bug in ES or this is a mix of features that shouldn't really not work together?
Is there an alternative to get the benefits of early termination but to be able to resort the results by different fields in a later stage?