Runtime field not appearing in search results

Elastic Stack Elasticsearch
1

I'm trying to set up a run time field to calculate the total of an e-commerce purchase. But there is something wrong with the way I'm declaring run time field, and it might have something to do with nested objects.

First I set up the index , the mapping and the runtime field like this:

PUT purchase
PUT purchase/_mapping
{
  "properties": {
    "items": {
      "type": "nested",
      "properties": {
        "product_id": {
          "type": "long"
        },
        "quantity": {
          "type": "long"
        },
        "product": {
          "properties": {
            "name": {
              "type": "keyword"
            },
            "product_id": {
              "type": "long"
            },
            "price": {
              "type": "long"
            }
          }
        }
      }
    },
    "purchase_date": {
      "type": "date"
    }
  },
  "runtime": {
    "items.total": {
      "type": "long",
      "script": {
        "source": "emit(doc['items.quantity'].value * doc['items.product.price'].value)"
      }
    }
  }
}

Then I post one test record lie this:

POST purchase/_doc
{
  "purchase_date": "2023-01-01",
  "items": [
    {
      "quantity": 2,
      "product_id": 1,
      "product": {
        "product_id": 1,
        "name": "pencil",
        "price": 1
      }
    },
    {
      "quantity": 4,
      "product_id": 2,
      "product": {
        "product_id": 2,
        "name": "stapler",
        "price": 5
      }
    }
  ]
}

When I do a GET purchase/_search, I get a result like this:

{
          "purchase_date": "2023-01-01",
          "items": [
            {
              "quantity": 2,
              "product_id": 1,
              "product": {
                "product_id": 1,
                "name": "pencil",
                "price": 1
              }
            },
            {
              "quantity": 4,
              "product_id": 2,
              "product": {
                "product_id": 2,
                "name": "stapler",
                "price": 5
              }
            }
          ]
        }

I was expecting a field order_items.total to be a sibling field to order_items.quantity. I've tried quite a few things but nothing seems to get the order_items.total field to appear.

What did I do wrong?

2

Hi @learningelastic

Read this answer, maybe dont work to nested fields.

Doc:

Use the fields parameter on the _search API to retrieve the values of runtime fields. Runtime fields won’t display in _source , but the fields API works for all fields, even those that were not sent as part of the original _source .

3

Thanks, you might be right that runtime fields don't work on nested objects. I just ran this search query:

GET purchase/_search
{
  "fields":["items.total"]
}

And I got this result:

{
  "error": {
    "root_cause": [
      {
        "type": "script_exception",
        "reason": "runtime error",
        "script_stack": [
          "org.elasticsearch.server@8.6.1/org.elasticsearch.index.fielddata.ScriptDocValues.throwIfEmpty(ScriptDocValues.java:92)",
          "org.elasticsearch.server@8.6.1/org.elasticsearch.index.fielddata.ScriptDocValues$Longs.get(ScriptDocValues.java:110)",
          "org.elasticsearch.server@8.6.1/org.elasticsearch.index.fielddata.ScriptDocValues$Longs.getValue(ScriptDocValues.java:105)",
          "emit(doc['items.quantity'].value * doc['items.product.price'].value)",
          "                          ^---- HERE"
        ],
        "script": "emit(doc['items.quantity'].value * doc['items.product.price'].value)",
        "lang": "painless",
        "position": {
          "offset": 26,
          "start": 0,
          "end": 68
        }
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "purchase",
        "node": "VYLFHFZ3RtOtR_P8dlnx3w",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "script_stack": [
            "org.elasticsearch.server@8.6.1/org.elasticsearch.index.fielddata.ScriptDocValues.throwIfEmpty(ScriptDocValues.java:92)",
            "org.elasticsearch.server@8.6.1/org.elasticsearch.index.fielddata.ScriptDocValues$Longs.get(ScriptDocValues.java:110)",
            "org.elasticsearch.server@8.6.1/org.elasticsearch.index.fielddata.ScriptDocValues$Longs.getValue(ScriptDocValues.java:105)",
            "emit(doc['items.quantity'].value * doc['items.product.price'].value)",
            "                          ^---- HERE"
          ],
          "script": "emit(doc['items.quantity'].value * doc['items.product.price'].value)",
          "lang": "painless",
          "position": {
            "offset": 26,
            "start": 0,
            "end": 68
          },
          "caused_by": {
            "type": "illegal_state_exception",
            "reason": "A document doesn't have a value for a field! Use doc[<field>].size()==0 to check if a document is missing a field!"
          }
        }
      }
    ]
  },
  "status": 400
}

If I change my mapping to delete the "type":"nested" line, then I run the same

GET purchase/_search
{
  "fields":["items.total"]
}

Then I do see

"fields": {
          "items.total": [
            2
          ]
        }

Which seems like elasticsearch just picked ONE record from the items array and multiplied product.price with the quantity.