How to use _source to create a scripted filter

wiouser · February 17, 2020, 3:02pm

Hi I am trying to write a query to return documents which has the length of the log field more than 900. It is a text field.
Running the below search I get variable _source is not defined

GET logdata/_search
{
  "query": {
    "bool": {
      "filter": {
        "script": {
          "script": {
            "source": "_source['log'].length() > 900",
            "lang": "painless"
          }
        }
      }
    }
  }
}

Any help appreciated.
Thanks

spinscale · February 17, 2020, 3:20pm

try something like doc['log.keyword'].value.length() (untested, on top of my head).

wiouser · February 20, 2020, 7:12am

Thanks for the reply. No, it doesn't work. I get the same error.
Tried in ver 6.7

spinscale · February 20, 2020, 7:48am

If you do not provide error messages it will be impossible to help further.

Please provide a fully reproducible, but minimal example including index creation with mappings, two sample documents (consisting only of that single field you are using in your query) being indexed and the query you are using, plus responses if they contain an error.

Thanks

wiouser · February 20, 2020, 8:06am

Hi,
Apologies, I thought it should be same for any text field. I have provided the details below.
I use the exact same query posted in the question

Document
{
"deviceid" : "XX12",
"time" : "2020-02-20T00:00:00.000Z",
"severity" : "notice",
"log" : "[system current time] : Thu Feb 20 00:00:00 UTC 2020",
}

Mapping

Summary

{
  "logdata" : {
    "mappings" : {
      "loginfo" : {
        "properties" : {
          "deviceid" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "log" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "severity" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "time" : {
            "type" : "date"
          }
        }
      }
    }
  }
}

Error message

Summary

{
"error": {
"root_cause": [
{
"type": "script_exception",
"reason": "compile error",
"script_stack": [
"_source['log'].va ...",
"^---- HERE"
],
"script": "_source['log'].value.length() > 10",
"lang": "painless"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "logdata",
"node": "xxxxxxxxxxxxxx",
"reason": {
"type": "query_shard_exception",
"reason": "failed to create query: {\n "bool" : {\n "filter" : [\n {\n "script" : {\n "script" : {\n "source" : "_source['log'].value.length() > 10",\n "lang" : "painless"\n },\n "boost" : 1.0\n }\n }\n ],\n "adjust_pure_negative" : true,\n "boost" : 1.0\n }\n}",
"index_uuid": "index_uuid",
"index": "logdata",
"caused_by": {
"type": "script_exception",
"reason": "compile error",
"script_stack": [
"_source['log'].va ...",
"^---- HERE"
],
"script": "_source['log'].value.length() > 10",
"lang": "painless",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Variable [_source] is not defined."
}
}
}
}
],
"caused_by": {
"type": "script_exception",
"reason": "compile error",
"script_stack": [
"_source['log'].va ...",
"^---- HERE"
],
"script": "_source['log'].value.length() > 10",
"lang": "painless",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Variable [_source] is not defined."
}
}
},
"status": 400
}

spinscale · February 20, 2020, 8:42am

your sample does not show my suggestion but still yours which returns an error.

wiouser · February 20, 2020, 9:55am

Please find the error messages for your suggestion below.
With keyword

Summary

{
  "error": {
    "root_cause": [
      {
        "type": "script_exception",
        "reason": "runtime error",
        "script_stack": [
          "doc['log.keyword'].value.length() > 10",
          "                        ^---- HERE"
        ],
        "script": "doc['log.keyword'].value.length() > 10",
        "lang": "painless"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "logdata",
        "node": "xxxxxxxxxxxxx",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "script_stack": [
            "doc['log.keyword'].value.length() > 10",
            "                        ^---- HERE"
          ],
          "script": "doc['log.keyword'].value.length() > 10",
          "lang": "painless",
          "caused_by": {
            "type": "null_pointer_exception",
            "reason": null
          }
        }
      }
    ]
  },
  "status": 500
}

Without keyword

Summary

{
  "error": {
    "root_cause": [
      {
        "type": "script_exception",
        "reason": "runtime error",
        "script_stack": [
          "org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:779)",
          "org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",
          "org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:283)",
          "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:88)",
          "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:85)",
          "java.security.AccessController.doPrivileged(Native Method)",
          "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:85)",
          "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:39)",
          "doc['log'].value.length() > 10",
          "    ^---- HERE"
        ],
        "script": "doc['log'].value.length() > 10",
        "lang": "painless"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "logdata",
        "node": "xxxxxxxxxxxx",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "script_stack": [
            "org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:779)",
            "org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",
            "org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:283)",
            "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:88)",
            "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:85)",
            "java.security.AccessController.doPrivileged(Native Method)",
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:85)",
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:39)",
            "doc['log'].value.length() > 10",
            "    ^---- HERE"
          ],
          "script": "doc['log'].value.length() > 10",
          "lang": "painless",
          "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [log] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
          }
        }
      }
    ]
  },
  "status": 500
}

spinscale · February 20, 2020, 10:32am

is it possible that not all of your documents have the log field set?

wiouser · February 20, 2020, 11:28am

Not possible. I checked again using exists query all documents have log field.

spinscale · February 20, 2020, 1:11pm

now helping is hard without concrete data. Can you create a minimal but reproducible example that includes index creation, mapping, two sample documents so that one can reproduce that issue?

Thanks!

wiouser · February 20, 2020, 1:31pm

Hi @spinscale

I have shared the document, search and mapping.
log.keyword doesn't work because it is not available in all documents. Like I said in my earlier replies it has ignore_above:256 in the mapping

So I created an test index with only one document.
Then, this works

But what I am interested is the log text field which is present in all the documents.
doc['log'].value.length doesn't work.

Error Message

{
"error": {
"root_cause": [
{
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:779)",
"org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",
"org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:283)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:88)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:85)",
"java.security.AccessController.doPrivileged(Native Method)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:85)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:39)",
"doc['log'].value.length() > 4",
" ^---- HERE"
],
"script": "doc['log'].value.length() > 4",
"lang": "painless"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "logdata",
"node": "E486K0eoRnyCBTJv7WBTrg",
"reason": {
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:779)",
"org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",
"org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:283)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:88)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:85)",
"java.security.AccessController.doPrivileged(Native Method)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:85)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:39)",
"doc['log'].value.length() > 4",
" ^---- HERE"
],
"script": "doc['log'].value.length() > 4",
"lang": "painless",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [log] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
}
}
}
]
},
"status": 500
}

What I am interested is to use the _source field to make this work. Is there any way to do this?

spinscale · February 20, 2020, 1:56pm

Hey,

so the _source field cannot be used, as this would mean, that the JSON source needs to be extracted for every hit, which is much slower than using doc values.

Try this instead

return doc['log.keyword'].size() > 0 && doc['log.keyword'].value.length() > 100

--Alex

system · March 19, 2020, 1:56pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
How to filter docs by text field length? Elasticsearch	2	9234	August 31, 2020
Variable [_source] is not defined Elasticsearch	5	4577	October 22, 2018
Filter Search by Scripted field elastic Elasticsearch	9	6328	March 10, 2020
_source variables in post_filter Elasticsearch	2	383	August 9, 2018
Filter be scripted field Elasticsearch	3	588	August 4, 2018

How to use _source to create a scripted filter

Related topics