Runtime fields not generated

Elastic Stack Kibana
1

We're trying to set up a runtime field for certain index or data stream that has the same field from another index. We've tried to run the following on the Dev Console and different alteration of it, but every attempt does not generate the field in Discover.

POST /"certain index"/_search
{
  "runtime_mappings": {
    "common.username": {
      "type": "keyword",
      "script": {
        "source": "if (doc['user.target.name'].size() != 0) { emit(doc['user.target.name'].value) }"
      }
    }
  },
  "query": {
    "match_all": {}
  },
  "fields": ["common.username"],
  "_source": ["user.target.name"]
}

Any insight would be greatly appreciated.

2

Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead.
Try this:

POST /"certain index"/_search
{
  "runtime_mappings": {
    "common.username": {
      "type": "keyword",
      "script": {
        "source": "if (doc['user.target.name.keyword'].size() != 0) { emit(doc['user.target.name.keyword'].value) }"
      }
    }
  },
  "query": {
    "match_all": {}
  },
  "fields": ["common.username"],
  "_source": ["user.target.name.keyword"]
}
3

Hi @jonathan.wong and welcome to the community!

There are several ways how runtime fields can be used:

  1. in _search request Define runtime fields in a search request | Elasticsearch Guide [8.15] | Elastic
  2. defined in mapping Map a runtime field | Elasticsearch Guide [8.15] | Elastic
  3. created in Kibana Data Views UI via "Create field" button

With (1) fields will not be persisted, only used during the search. To actually create and persist runtime fields, consider using (2) or (3).

4

Hi Shanker,

Unfortunately the posted code did not work. Below is the output of the error message.

{
  "error": {
    "root_cause": [
      {
        "type": "script_exception",
        "reason": "runtime error",
        "script_stack": [
          "org.elasticsearch.server@8.15.0/org.elasticsearch.search.lookup.LeafDocLookup.getFactoryForDoc(LeafDocLookup.java:145)",
          "org.elasticsearch.server@8.15.0/org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:185)",
          "org.elasticsearch.server@8.15.0/org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:32)",
          "if (doc['user.target.name.keyword'].size() != 0) { ",
          "        ^---- HERE"
        ],
        "script": "if (doc['user.target.name.keyword'].size() != 0) { emit(doc['user.target.name.keyword'].value) }",
        "lang": "painless",
        "position": {
          "offset": 8,
          "start": 0,
          "end": 51
        }
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": ".ds-logs-system.security-default-2024.08.19-000131",
        "node": "tF6HvkxyRSOx7X4_TM6_mg",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "script_stack": [
            "org.elasticsearch.server@8.15.0/org.elasticsearch.search.lookup.LeafDocLookup.getFactoryForDoc(LeafDocLookup.java:145)",
            "org.elasticsearch.server@8.15.0/org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:185)",
            "org.elasticsearch.server@8.15.0/org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:32)",
            "if (doc['user.target.name.keyword'].size() != 0) { ",
            "        ^---- HERE"
          ],
          "script": "if (doc['user.target.name.keyword'].size() != 0) { emit(doc['user.target.name.keyword'].value) }",
          "lang": "painless",
          "position": {
            "offset": 8,
            "start": 0,
            "end": 51
          },
          "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "No field found for [user.target.name.keyword] in mapping"
          }
        }
      }
    ]
  },
  "status": 400
}
5

Hi Julia,

We are attempting to perform this as option one to ensure the painless script executes correctly before defining a mapping a runtime field.