# Index Sorting and terminate\_after combination

**URL:** <https://discuss.elastic.co/t/index-sorting-and-terminate-after-combination/198233>\
**Category:** Elasticsearch\
**Created:** [September 5, 2019, 11:18am UTC](https://discuss.elastic.co/t/index-sorting-and-terminate-after-combination/198233 "2019-09-05T11:18:59Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![HenriqueLimao](https://avatars.discourse-cdn.com/v4/letter/h/65b543/32.png) [@HenriqueLimao](https://discuss.elastic.co/u/HenriqueLimao)\
**Post date:** [September 5, 2019, 11:18am UTC](https://discuss.elastic.co/t/index-sorting-and-terminate-after-combination/198233/1 "2019-09-05T11:18:59Z")

</div>

I have an index in ES 6.6 used to search regular content that should be scored differently based on the user. Today we have more than 50 fields that can be used to calculate this score.

In order to speed up the query, I decided to use early termination like described [here](https://www.elastic.co/guide/en/elasticsearch/reference/master/index-modules-index-sorting.html)

I created a field based on a combination of all individual fields used for scoring and sorted the index with this value.  
However, I can't sort the results of the query based on this single field (since the sort order will depend on each user), so I started using the `terminate_after` parameter in the query.  
After doing some evaluations I noticed that terminating the query after 1000 elements was good enough to have a set of 10 results with the optimal score for each user.

This seemed to work well and I got the speed up that I wanted, however later I realized that this only works if each shard in the index has only one segment. I call force\_merge with value 1 during index creation, but I can't do that for index updates during normal traffic.

After updating the index, new segments are created and they are completely ignored by the query if we reach enough matches for the `terminate_after` parameter in the original segment.  
This happen even if in the new segment, the field that defines the index sorting would position the document as the top result.

My questions now:

- Is that a bug in ES or this is a mix of features that shouldn't really not work together?

- Is there an alternative to get the benefits of early termination but to be able to resort the results by different fields in a later stage?

---

<div class="post-metadata">

**Author:** ![HenriqueLimao](https://avatars.discourse-cdn.com/v4/letter/h/65b543/32.png) [@HenriqueLimao](https://discuss.elastic.co/u/HenriqueLimao)\
**Post date:** [September 5, 2019, 11:48am UTC](https://discuss.elastic.co/t/index-sorting-and-terminate-after-combination/198233/2 "2019-09-05T11:48:42Z")

</div>

I put together a script that can be executed in kibana to give an example of the issue:

```auto
PUT the-index
{
  "settings": {
    "index": {
      "number_of_shards": 1,
      "number_of_replicas": 1
    },
    "sort": {
      "field": "combined_score",
      "order": "desc"
    }
  },
  "mappings": {
    "doc": {
      "properties": {
        "content": { "type": "text" },
        "score_field_1": { "type": "integer" },
        "score_field_2": { "type": "integer" },
        "score_field_3": { "type": "integer" },
        "combined_score": { "type": "integer" }
      }
    }
  }
}

PUT /the-index/doc/id1
{
  "content": "content 1",
  "score_field_1": 10,
  "score_field_2": 40,
  "score_field_3": 100,
  "combined_score": 150
}

PUT /the-index/doc/id2
{
  "content": "content 2",
  "score_field_1": 90,
  "score_field_2": 40,
  "score_field_3": 10,
  "combined_score": 140
}

PUT /the-index/doc/id3
{
  "content": "content 3",
  "score_field_1": 80,
  "score_field_2": 40,
  "score_field_3": 10,
  "combined_score": 130
}

PUT /the-index/doc/id4
{
  "content": "content 4",
  "score_field_1": 70,
  "score_field_2": 40,
  "score_field_3": 10,
  "combined_score": 120
}

PUT /the-index/doc/id5
{
  "content": "content 5",
  "score_field_1": 60,
  "score_field_2": 40,
  "score_field_3": 10,
  "combined_score": 110
}

GET /the-index/_segments

POST /the-index/_forcemerge?max_num_segments=1

#returns correct result id1 as top hit
GET /the-index/_search
{
  "terminate_after": 2,
  "size": 1,
  "query": {
    "function_score": {
      "query": {
        "match": {
          "content": "content"
        }
      },
      "field_value_factor": {
        "field": "score_field_3"
      }
    }
  }
}

# index id1 again, this will create a new segment
PUT /the-index/doc/id1
{
  "content": "content 1",
  "score_field_1": 10,
  "score_field_2": 40,
  "score_field_3": 100,
  "combined_score": 150
}

#returns incorrect result id2 now as top hit
GET /the-index/_search
{
  "terminate_after": 2,
  "size": 1,
  "query": {
    "function_score": {
      "query": {
        "match": {
          "content": "content"
        }
      },
      "field_value_factor": {
        "field": "score_field_3"
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 3, 2019, 11:48am UTC](https://discuss.elastic.co/t/index-sorting-and-terminate-after-combination/198233/3 "2019-10-03T11:48:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

---

<div class="post-metadata">

**Author:** ![jimczi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jimczi/32/47985_2.png) [@jimczi](https://discuss.elastic.co/u/jimczi)\
**Post date:** [January 21, 2020, 3:25pm UTC](https://discuss.elastic.co/t/index-sorting-and-terminate-after-combination/198233/4 "2020-01-21T15:25:04Z")

</div>

> [@HenriqueLimao](#):
>
> Is that a bug in ES or this is a mix of features that shouldn't really not work together?

The query phase visits each segment independently but `terminate_after` is a global count per shard so there's no guarantee that you'll see a sorted stream even if the index is sorted. As you already noticed it works if you force merge all shards to 1 segment but that's just a side-effect of the fact that the index is sorted, not a bug since `terminate_after` does not change the way documents are collected.  
Today the only way to achieve what you want would be to retrieve an extended top N sorted by the index sort criteria and to sort/prune this result client-side using the value of a field extracted from the \_source or from the doc values. This is not ideal so I think it's worth opening a feature request in github to discuss a full solution that would not require extra logic in the client. Could you open one ?