When I use scripted field and error show "runtime error" and "illegal_argument_exception"

beam5544 · October 29, 2020, 7:00am

When I use this script

doc['reqBody'].length

And I go to "Discover" page to show that field, and this happened.

{
  "took": 201,
  "timed_out": false,
  "num_reduce_phases": 92,
  "_shards": {
    "total": 672,
    "successful": 656,
    "skipped": 306,
    "failed": 16,
    "failures": [
      {
        "shard": 2,
        "index": "cpx-request-stream-log-ms-current-2020.10.28",
        "node": "71QJmoFbRE6gUqt32ulOMg",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "script_stack": [
            "org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:763)",
            "org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",
            "org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:330)",
            "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:97)",
            "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:94)",
            "java.base/java.security.AccessController.doPrivileged(AccessController.java:312)",
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:94)",
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)",
            "doc['reqBody'].length",
            "    ^---- HERE"
          ],
          "script": "doc['reqBody'].length",
          "lang": "painless",
          "position": {
            "offset": 4,
            "start": 0,
            "end": 21
          },
          "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [reqBody] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
          }
        }
      },
      {
        "shard": 6,
        "index": "cpx-request-stream-log-ms-current-2020.10.27",
        "node": "71QJmoFbRE6gUqt32ulOMg",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "script_stack": [
            "org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:763)",
            "org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",
            "org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:330)",
            "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:97)",
            "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:94)",
            "java.base/java.security.AccessController.doPrivileged(AccessController.java:312)",
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:94)",
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)",
            "doc['reqBody'].length",
            "    ^---- HERE"
          ],
          "script": "doc['reqBody'].length",
          "lang": "painless",
          "position": {
            "offset": 4,
            "start": 0,
            "end": 21
          },
          "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [reqBody] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
          }
        }
      },
      {
        "shard": 11,
        "index": "cpx-request-stream-log-ms-current-2020.10.26",
        "node": "IY2aR2zaQ2Wp1HdX2EXs4w",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "script_stack": [
            "org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:763)",
            "org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",
            "org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:330)",
            "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:97)",
            "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:94)",
            "java.base/java.security.AccessController.doPrivileged(AccessController.java:312)",
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:94)",
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)",
            "doc['reqBody'].length",
            "    ^---- HERE"
          ],
          "script": "doc['reqBody'].length",
          "lang": "painless",
          "position": {
            "offset": 4,
            "start": 0,
            "end": 21
          },
          "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [reqBody] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
          }
        }
      }
    ]
  },
  "hits": {
    "total": 23,
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "2": {
      "buckets": [
        {
          "key_as_string": "2020-10-26T09:00:00.000+07:00",
          "key": 1603677600000,
          "doc_count": 3
        },
        {
          "key_as_string": "2020-10-26T15:00:00.000+07:00",
          "key": 1603699200000,
          "doc_count": 6
        },
        {
          "key_as_string": "2020-10-27T09:00:00.000+07:00",
          "key": 1603764000000,
          "doc_count": 2
        },
        {
          "key_as_string": "2020-10-27T12:00:00.000+07:00",
          "key": 1603774800000,
          "doc_count": 6
        },
        {
          "key_as_string": "2020-10-28T09:00:00.000+07:00",
          "key": 1603850400000,
          "doc_count": 4
        },
        {
          "key_as_string": "2020-10-28T12:00:00.000+07:00",
          "key": 1603861200000,
          "doc_count": 1
        },
        {
          "key_as_string": "2020-10-28T15:00:00.000+07:00",
          "key": 1603872000000,
          "doc_count": 1
        }
      ]
    }
  }
}

What I need to fix it ?

Sample data of 'reqBody'

{"masterStagingId":"PAYROLL72e6972d84b047af88e62258dbaea998","paymentProductId":"PAYROLL","batchReferenceId":"PAY201028df78fa27a86","scheduleMainId":null,"scheduleDetailId":null,"isLastSchedule":null,"batchStatus":"PROCESSING","isBatchStatusFinal":false,"isDebited":false}

flash1293 · October 29, 2020, 8:08am

reqBody is indexed as a text field, but scripted fields are operating on the fielddata. Either use a keyword indexed field in your script (if you use the default mapping that would be reqBody.keyword or enable fielddata for this field in your mapping like the error message suggests (if you enable fielddata, you need to reindex your data for it to take effect for existing data)

beam5544 · October 29, 2020, 10:41am

How to enable fielddata that you suggest me ? I think this is only way for me cuz now I can't find any field that have .keyword after field name.

beam5544 · October 29, 2020, 10:48am

And pls give me some example to revert it back if this don't work XD ... im so scared of that.

flash1293 · October 30, 2020, 7:13am

To reindex you don't have to overwrite your existing data, you can set it up side by side. As a first step, you define the updated mapping with fielddata enabled in a separate place, then shovel the data over from your current index using the reindex API: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html

However, looking at your other question I think that's not actually the right way for your case. It looks like you have some JSON which is indexed as a string. Can you explain in greater detail what you are trying to achieve? It seems like there is a better way.

beam5544 · November 2, 2020, 3:23am

For the reqBody that is JSON which is indexed as a string, I only want some "key" in that JSON for query such as, I need batchStatus for query because batchStatus can be PROCESSING, SUCCESS, FAILED that can show in bar chart or sth.

flash1293 · November 2, 2020, 7:34am

You can probably achieve that using a scripted field, but it will be complex, brittle and really slow. The recommended way to do this is to ingest the JSON string as a real object in the first place, so Elasticsearch knows all the sub-properties and can put them into proper indices which will be super fast to query.

I don't know how you are ingesting data, but take a look at ingest pipelines (https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html) and the JSON processor https://www.elastic.co/guide/en/elasticsearch/reference/current/json-processor.html or alternatively Logstash along with the JSON filter (https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html)

beam5544 · November 4, 2020, 10:26am

@flash1293

Thanks for your helping !

system · December 2, 2020, 10:26am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Script exception runtime error Kibana painless	4	2113	June 27, 2022
[script_exception] runtime error bad request Kibana painless	2	1050	August 5, 2020
Error with scriptfield in visualization Kibana	4	968	April 6, 2020
[script_exception] runtime error in Discover Kibana	6	3420	November 13, 2020
Using scripted fields from Kibana raises a script exception in ES Elasticsearch	3	1513	July 5, 2017

When I use scripted field and error show "runtime error" and "illegal_argument_exception"

Related topics