Nested sorting differs between ES7 and ES8?

[MarynaCherniavska]

Dear team, we're moving from 7.17 to 8.7 (I know, big jump) and as part of the move, we have been running the e2e tests of the application on the new cluster, before actually switching the application to it. Some tests, however, started failing. We traced these to requests where a sorting was happening on a nested field.

Specifically, the error we're getting is as follows:

{
        "type": "query_shard_exception",
        "reason": "it is mandatory to set the [nested] context on the nested sort field: [trace.origami.timestamp].",
        "index_uuid": "_xvEa8gNSFyCDm0aFXqYhg",
        "index": "article_ad1"
      }

This seems to refer to the sort clause that we have:

"sort": [
      {
        "trace.origami.timestamp": {
          "order": "desc",
          "missing": "_last",
          "unmapped_type": "float",
          "mode": "min"
        }
      }
    ]

The page about sorting on a nested field for ES 8.8 (current) says that there should be a path specified in a "nested.path" clause of the sort. However, the same page for ES 7.17 states exactly the same, but the query still runs fine without that clause! I know how to fix the query to make it work. What I don't understand is, what changed between the versions so that it started erroring out in ES8, whereas in ES7 it was working fine, despite stating in the doc that the parameter is optional?

I've prepared a small test case to reproduce as close as possible to what we're having.

ES_PATH=http://localhost:9200
TEST_INDEX=test-index

curl -X PUT "$ES_PATH/$TEST_INDEX"

curl -X POST --url "$ES_PATH/$TEST_INDEX/_mapping" -H "Content-Type:application/json" -d '{
  "properties": {
    "name": {
      "type": "text"
    },
    "trace": {
      "type": "nested",
      "properties": {
        "origami": {
          "properties": {
            "timestamp": {
              "type": "long"
            }
          }
        }
      }
    }
  }
}'

curl -X POST --url "$ES_PATH/$TEST_INDEX/_bulk" -H "Content-Type:application/json" -d '
{ "index" : { "_id" : "1" } }
{ "trace" : { "origami": { "timestamp": 1688732801 } } }
{ "index" : { "_id" : "2" } }
{ "trace" : { "origami": { "timestamp": 1688732701 } } }
{ "index" : { "_id" : "3" } }
{ "trace" : { "origami": { "timestamp": 1688732601 } } }
'

# this succeeds in both ES7 and ES8

curl -X POST --url "$ES_PATH/$TEST_INDEX/_search" -H "Content-Type:application/json" -d '
{
  "query": {
    "match_all": {}
  },
  "sort": [
      {
        "trace.origami.timestamp": {
          "order": "asc",
          "missing": "_last",
          "unmapped_type": "float",
          "mode": "min",
          "nested": {
            "path": "trace"
          }
        }
      }
    ]
}' | jq

# this fails in ES8 but works in ES7

curl -X POST --url "$ES_PATH/$TEST_INDEX/_search" -H "Content-Type:application/json" -d '
{
  "query": {
    "match_all": {}
  },
  "sort": [
      {
        "trace.origami.timestamp": {
          "order": "asc",
          "missing": "_last",
          "unmapped_type": "float",
          "mode": "min"
        }
      }
    ]
}' | jq

[dadoonet (David Pilato)]

Thanks for the detailed script. I check it and you're right, it's ok in 7.17.12 but fails in 8.x.

I believe that in 7.17 there was may be a bug which has been fixed in 8?

BTW, I updated and simplified your script. In case someone else wants to test/comment it:

DELETE /test-index
PUT /test-index
{
  "mappings": {
    "properties": {
      "name": {
        "type": "text"
      },
      "trace": {
        "type": "nested",
        "properties": {
          "origami": {
            "properties": {
              "timestamp": {
                "type": "long"
              }
            }
          }
        }
      }
    }
  }
}

POST /test-index/_bulk
{ "index" : { "_id" : "1" } }
{ "trace" : { "origami": { "timestamp": 1688732801 } } }
{ "index" : { "_id" : "2" } }
{ "trace" : { "origami": { "timestamp": 1688732701 } } }
{ "index" : { "_id" : "3" } }
{ "trace" : { "origami": { "timestamp": 1688732601 } } }


GET /test-index/_search
{
  "sort": [
    {
      "trace.origami.timestamp": {
        "nested": {
          "path": "trace"
        }
      }
    }
  ]
}

# this fails in ES8 but works in ES7

GET /test-index/_search
{
  "sort": [
    {
      "trace.origami.timestamp": {
        "nested": {
          "path": "trace"
        }
      }
    }
  ]
}