When I use scripted field and error show "runtime error" and "illegal_argument_exception"

Elastic Stack Kibana
1

When I use this script

doc['reqBody'].length

And I go to "Discover" page to show that field, and this happened.

{
  "took": 201,
  "timed_out": false,
  "num_reduce_phases": 92,
  "_shards": {
    "total": 672,
    "successful": 656,
    "skipped": 306,
    "failed": 16,
    "failures": [
      {
        "shard": 2,
        "index": "cpx-request-stream-log-ms-current-2020.10.28",
        "node": "71QJmoFbRE6gUqt32ulOMg",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "script_stack": [
            "org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:763)",
            "org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",
            "org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:330)",
            "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:97)",
            "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:94)",
            "java.base/java.security.AccessController.doPrivileged(AccessController.java:312)",
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:94)",
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)",
            "doc['reqBody'].length",
            "    ^---- HERE"
          ],
          "script": "doc['reqBody'].length",
          "lang": "painless",
          "position": {
            "offset": 4,
            "start": 0,
            "end": 21
          },
          "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [reqBody] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
          }
        }
      },
      {
        "shard": 6,
        "index": "cpx-request-stream-log-ms-current-2020.10.27",
        "node": "71QJmoFbRE6gUqt32ulOMg",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "script_stack": [
            "org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:763)",
            "org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",
            "org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:330)",
            "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:97)",
            "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:94)",
            "java.base/java.security.AccessController.doPrivileged(AccessController.java:312)",
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:94)",
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)",
            "doc['reqBody'].length",
            "    ^---- HERE"
          ],
          "script": "doc['reqBody'].length",
          "lang": "painless",
          "position": {
            "offset": 4,
            "start": 0,
            "end": 21
          },
          "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [reqBody] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
          }
        }
      },
      {
        "shard": 11,
        "index": "cpx-request-stream-log-ms-current-2020.10.26",
        "node": "IY2aR2zaQ2Wp1HdX2EXs4w",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "script_stack": [
            "org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:763)",
            "org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",
            "org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:330)",
            "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:97)",
            "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:94)",
            "java.base/java.security.AccessController.doPrivileged(AccessController.java:312)",
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:94)",
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)",
            "doc['reqBody'].length",
            "    ^---- HERE"
          ],
          "script": "doc['reqBody'].length",
          "lang": "painless",
          "position": {
            "offset": 4,
            "start": 0,
            "end": 21
          },
          "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [reqBody] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
          }
        }
      }
    ]
  },
  "hits": {
    "total": 23,
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "2": {
      "buckets": [
        {
          "key_as_string": "2020-10-26T09:00:00.000+07:00",
          "key": 1603677600000,
          "doc_count": 3
        },
        {
          "key_as_string": "2020-10-26T15:00:00.000+07:00",
          "key": 1603699200000,
          "doc_count": 6
        },
        {
          "key_as_string": "2020-10-27T09:00:00.000+07:00",
          "key": 1603764000000,
          "doc_count": 2
        },
        {
          "key_as_string": "2020-10-27T12:00:00.000+07:00",
          "key": 1603774800000,
          "doc_count": 6
        },
        {
          "key_as_string": "2020-10-28T09:00:00.000+07:00",
          "key": 1603850400000,
          "doc_count": 4
        },
        {
          "key_as_string": "2020-10-28T12:00:00.000+07:00",
          "key": 1603861200000,
          "doc_count": 1
        },
        {
          "key_as_string": "2020-10-28T15:00:00.000+07:00",
          "key": 1603872000000,
          "doc_count": 1
        }
      ]
    }
  }
}

What I need to fix it ?

Sample data of 'reqBody'

{"masterStagingId":"PAYROLL72e6972d84b047af88e62258dbaea998","paymentProductId":"PAYROLL","batchReferenceId":"PAY201028df78fa27a86","scheduleMainId":null,"scheduleDetailId":null,"isLastSchedule":null,"batchStatus":"PROCESSING","isBatchStatusFinal":false,"isDebited":false}
2

reqBody is indexed as a text field, but scripted fields are operating on the fielddata. Either use a keyword indexed field in your script (if you use the default mapping that would be reqBody.keyword or enable fielddata for this field in your mapping like the error message suggests (if you enable fielddata, you need to reindex your data for it to take effect for existing data)

3

How to enable fielddata that you suggest me ? I think this is only way for me cuz now I can't find any field that have .keyword after field name.

4

And pls give me some example to revert it back if this don't work XD ... im so scared of that.

5

To reindex you don't have to overwrite your existing data, you can set it up side by side. As a first step, you define the updated mapping with fielddata enabled in a separate place, then shovel the data over from your current index using the reindex API: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html

However, looking at your other question I think that's not actually the right way for your case. It looks like you have some JSON which is indexed as a string. Can you explain in greater detail what you are trying to achieve? It seems like there is a better way.

6

For the reqBody that is JSON which is indexed as a string, I only want some "key" in that JSON for query such as, I need batchStatus for query because batchStatus can be PROCESSING, SUCCESS, FAILED that can show in bar chart or sth.

7

You can probably achieve that using a scripted field, but it will be complex, brittle and really slow. The recommended way to do this is to ingest the JSON string as a real object in the first place, so Elasticsearch knows all the sub-properties and can put them into proper indices which will be super fast to query.

I don't know how you are ingesting data, but take a look at ingest pipelines (https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html) and the JSON processor https://www.elastic.co/guide/en/elasticsearch/reference/current/json-processor.html or alternatively Logstash along with the JSON filter (https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html)

8

@flash1293

Thanks for your helping !

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.