# How to use \_source to create a scripted filter

**URL:** <https://discuss.elastic.co/t/how-to-use-source-to-create-a-scripted-filter/219646>\
**Category:** Elasticsearch\
**Created:** [February 17, 2020, 3:02pm UTC](https://discuss.elastic.co/t/how-to-use-source-to-create-a-scripted-filter/219646 "2020-02-17T15:02:33Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![wiouser](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wiouser/32/45741_2.png) [@wiouser](https://discuss.elastic.co/u/wiouser)\
**Post date:** [February 17, 2020, 3:02pm UTC](https://discuss.elastic.co/t/how-to-use-source-to-create-a-scripted-filter/219646/1 "2020-02-17T15:02:34Z")

</div>

Hi I am trying to write a query to return documents which has the length of the `log` field more than 900. It is a `text` field.  
Running the below search I get `variable _source is not defined`

```
GET logdata/_search
{
  "query": {
    "bool": {
      "filter": {
        "script": {
          "script": {
            "source": "_source['log'].length() > 900",
            "lang": "painless"
          }
        }
      }
    }
  }
}

```

Any help appreciated.  
Thanks

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [February 17, 2020, 3:20pm UTC](https://discuss.elastic.co/t/how-to-use-source-to-create-a-scripted-filter/219646/2 "2020-02-17T15:20:12Z")

</div>

try something like `doc['log.keyword'].value.length()` (untested, on top of my head).

---

<div class="post-metadata">

**Author:** ![wiouser](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wiouser/32/45741_2.png) [@wiouser](https://discuss.elastic.co/u/wiouser)\
**Post date:** [February 20, 2020, 7:12am UTC](https://discuss.elastic.co/t/how-to-use-source-to-create-a-scripted-filter/219646/3 "2020-02-20T07:12:27Z")

</div>

Thanks for the reply. No, it doesn't work. I get the same error.  
Tried in ver 6.7

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [February 20, 2020, 7:48am UTC](https://discuss.elastic.co/t/how-to-use-source-to-create-a-scripted-filter/219646/4 "2020-02-20T07:48:51Z")

</div>

If you do not provide error messages it will be impossible to help further.

Please provide a fully reproducible, but **minimal** example including index creation with mappings, two sample documents (consisting only of that single field you are using in your query) being indexed and the query you are using, plus responses if they contain an error.

Thanks

---

<div class="post-metadata">

**Author:** ![wiouser](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wiouser/32/45741_2.png) [@wiouser](https://discuss.elastic.co/u/wiouser)\
**Post date:** [February 20, 2020, 8:06am UTC](https://discuss.elastic.co/t/how-to-use-source-to-create-a-scripted-filter/219646/5 "2020-02-20T08:06:37Z")

</div>

Hi,  
Apologies, I thought it should be same for any text field. I have provided the details below.  
I use the exact same query posted in the question

Document  
{  
"deviceid" : "XX12",  
"time" : "2020-02-20T00:00:00.000Z",  
"severity" : "notice",  
"log" : "[system current time] : Thu Feb 20 00:00:00 UTC 2020",  
}

Mapping

> **Summary**
>
> ```
> {
> "logdata" : {
> "mappings" : {
> "loginfo" : {
> "properties" : {
> "deviceid" : {
> "type" : "text",
> "fields" : {
> "keyword" : {
> "type" : "keyword",
> "ignore_above" : 256
> }
> }
> },
> "log" : {
> "type" : "text",
> "fields" : {
> "keyword" : {
> "type" : "keyword",
> "ignore_above" : 256
> }
> }
> },
> "severity" : {
> "type" : "text",
> "fields" : {
> "keyword" : {
> "type" : "keyword",
> "ignore_above" : 256
> }
> }
> },
> "time" : {
> "type" : "date"
> }
> }
> }
> }
> }
> }
> 
> ```

Error message

> **Summary**
>
> {  
> "error": {  
> "root\_cause": [  
> {  
> "type": "script\_exception",  
> "reason": "compile error",  
> "script\_stack": [  
> "\_source['log'].va ...",  
> "^---- HERE"  
> ],  
> "script": "\_source['log'].value.length() \> 10",  
> "lang": "painless"  
> }  
> ],  
> "type": "search\_phase\_execution\_exception",  
> "reason": "all shards failed",  
> "phase": "query",  
> "grouped": true,  
> "failed\_shards": [  
> {  
> "shard": 0,  
> "index": "logdata",  
> "node": "xxxxxxxxxxxxxx",  
> "reason": {  
> "type": "query\_shard\_exception",  
> "reason": "failed to create query: {\n "bool" : {\n "filter" : [\n {\n "script" : {\n "script" : {\n "source" : "\_source['log'].value.length() \> 10",\n "lang" : "painless"\n },\n "boost" : 1.0\n }\n }\n ],\n "adjust\_pure\_negative" : true,\n "boost" : 1.0\n }\n}",  
> "index\_uuid": "index\_uuid",  
> "index": "logdata",  
> "caused\_by": {  
> "type": "script\_exception",  
> "reason": "compile error",  
> "script\_stack": [  
> "\_source['log'].va ...",  
> "^---- HERE"  
> ],  
> "script": "\_source['log'].value.length() \> 10",  
> "lang": "painless",  
> "caused\_by": {  
> "type": "illegal\_argument\_exception",  
> "reason": "Variable [\_source] is not defined."  
> }  
> }  
> }  
> }  
> ],  
> "caused\_by": {  
> "type": "script\_exception",  
> "reason": "compile error",  
> "script\_stack": [  
> "\_source['log'].va ...",  
> "^---- HERE"  
> ],  
> "script": "\_source['log'].value.length() \> 10",  
> "lang": "painless",  
> "caused\_by": {  
> "type": "illegal\_argument\_exception",  
> "reason": "Variable [\_source] is not defined."  
> }  
> }  
> },  
> "status": 400  
> }

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [February 20, 2020, 8:42am UTC](https://discuss.elastic.co/t/how-to-use-source-to-create-a-scripted-filter/219646/6 "2020-02-20T08:42:29Z")

</div>

> [@wiouser](#):
>
> \_source['log'].value.length() \> 10

your sample does not show my suggestion but still yours which returns an error.

---

<div class="post-metadata">

**Author:** ![wiouser](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wiouser/32/45741_2.png) [@wiouser](https://discuss.elastic.co/u/wiouser)\
**Post date:** [February 20, 2020, 9:55am UTC](https://discuss.elastic.co/t/how-to-use-source-to-create-a-scripted-filter/219646/7 "2020-02-20T09:55:57Z")

</div>

Please find the error messages for your suggestion below.  
With keyword

> **Summary**
>
> ```
> {
> "error": {
> "root_cause": [
> {
> "type": "script_exception",
> "reason": "runtime error",
> "script_stack": [
> "doc['log.keyword'].value.length() > 10",
> " ^---- HERE"
> ],
> "script": "doc['log.keyword'].value.length() > 10",
> "lang": "painless"
> }
> ],
> "type": "search_phase_execution_exception",
> "reason": "all shards failed",
> "phase": "query",
> "grouped": true,
> "failed_shards": [
> {
> "shard": 0,
> "index": "logdata",
> "node": "xxxxxxxxxxxxx",
> "reason": {
> "type": "script_exception",
> "reason": "runtime error",
> "script_stack": [
> "doc['log.keyword'].value.length() > 10",
> " ^---- HERE"
> ],
> "script": "doc['log.keyword'].value.length() > 10",
> "lang": "painless",
> "caused_by": {
> "type": "null_pointer_exception",
> "reason": null
> }
> }
> }
> ]
> },
> "status": 500
> }
> 
> ```

Without keyword

> **Summary**
>
> ```
> {
> "error": {
> "root_cause": [
> {
> "type": "script_exception",
> "reason": "runtime error",
> "script_stack": [
> "org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:779)",
> "org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",
> "org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:283)",
> "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:88)",
> "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:85)",
> "java.security.AccessController.doPrivileged(Native Method)",
> "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:85)",
> "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:39)",
> "doc['log'].value.length() > 10",
> " ^---- HERE"
> ],
> "script": "doc['log'].value.length() > 10",
> "lang": "painless"
> }
> ],
> "type": "search_phase_execution_exception",
> "reason": "all shards failed",
> "phase": "query",
> "grouped": true,
> "failed_shards": [
> {
> "shard": 0,
> "index": "logdata",
> "node": "xxxxxxxxxxxx",
> "reason": {
> "type": "script_exception",
> "reason": "runtime error",
> "script_stack": [
> "org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:779)",
> "org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",
> "org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:283)",
> "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:88)",
> "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:85)",
> "java.security.AccessController.doPrivileged(Native Method)",
> "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:85)",
> "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:39)",
> "doc['log'].value.length() > 10",
> " ^---- HERE"
> ],
> "script": "doc['log'].value.length() > 10",
> "lang": "painless",
> "caused_by": {
> "type": "illegal_argument_exception",
> "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [log] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
> }
> }
> }
> ]
> },
> "status": 500
> }
> 
> ```

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [February 20, 2020, 10:32am UTC](https://discuss.elastic.co/t/how-to-use-source-to-create-a-scripted-filter/219646/8 "2020-02-20T10:32:59Z")

</div>

is it possible that not all of your documents have the `log` field set?

---

<div class="post-metadata">

**Author:** ![wiouser](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wiouser/32/45741_2.png) [@wiouser](https://discuss.elastic.co/u/wiouser)\
**Post date:** [February 20, 2020, 11:28am UTC](https://discuss.elastic.co/t/how-to-use-source-to-create-a-scripted-filter/219646/9 "2020-02-20T11:28:15Z")

</div>

Not possible. I checked again using `exists` query all documents have log field.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [February 20, 2020, 1:11pm UTC](https://discuss.elastic.co/t/how-to-use-source-to-create-a-scripted-filter/219646/10 "2020-02-20T13:11:19Z")

</div>

now helping is hard without concrete data. Can you create a **minimal but reproducible example** that includes index creation, mapping, two sample documents so that one can reproduce that issue?

Thanks!

---

<div class="post-metadata">

**Author:** ![wiouser](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wiouser/32/45741_2.png) [@wiouser](https://discuss.elastic.co/u/wiouser)\
**Post date:** [February 20, 2020, 1:31pm UTC](https://discuss.elastic.co/t/how-to-use-source-to-create-a-scripted-filter/219646/11 "2020-02-20T13:31:28Z")

</div>

Hi @spinscale

I have shared the document, search and mapping.  
`log.keyword` doesn't work because it is not available in all documents. Like I said in my earlier replies it has `ignore_above:256` in the mapping

So I created an test index with only one document.  
Then, this works

> [@spinscale](#):
>
> try something like `doc['log.keyword'].value.length()` (untested, on top of my head).

But what I am interested is the `log` text field which is present in **all** the documents.  
_doc[' **log**'].value.length_ doesn't work.

> **Error Message**
>
> {  
> "error": {  
> "root\_cause": [  
> {  
> "type": "script\_exception",  
> "reason": "runtime error",  
> "script\_stack": [  
> "org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:779)",  
> "org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",  
> "org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:283)",  
> "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:88)",  
> "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:85)",  
> "java.security.AccessController.doPrivileged(Native Method)",  
> "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:85)",  
> "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:39)",  
> "doc['log'].value.length() \> 4",  
> " ^---- HERE"  
> ],  
> "script": "doc['log'].value.length() \> 4",  
> "lang": "painless"  
> }  
> ],  
> "type": "search\_phase\_execution\_exception",  
> "reason": "all shards failed",  
> "phase": "query",  
> "grouped": true,  
> "failed\_shards": [  
> {  
> "shard": 0,  
> "index": "logdata",  
> "node": "E486K0eoRnyCBTJv7WBTrg",  
> "reason": {  
> "type": "script\_exception",  
> "reason": "runtime error",  
> "script\_stack": [  
> "org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:779)",  
> "org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",  
> "org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:283)",  
> "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:88)",  
> "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:85)",  
> "java.security.AccessController.doPrivileged(Native Method)",  
> "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:85)",  
> "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:39)",  
> "doc['log'].value.length() \> 4",  
> " ^---- HERE"  
> ],  
> "script": "doc['log'].value.length() \> 4",  
> "lang": "painless",  
> "caused\_by": {  
> "type": "illegal\_argument\_exception",  
> "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [log] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."  
> }  
> }  
> }  
> ]  
> },  
> "status": 500  
> }

What I am interested is to use the _\_source_ field to make this work. Is there any way to do this?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [February 20, 2020, 1:56pm UTC](https://discuss.elastic.co/t/how-to-use-source-to-create-a-scripted-filter/219646/12 "2020-02-20T13:56:28Z")

</div>

Hey,

so the \_source field cannot be used, as this would mean, that the JSON source needs to be extracted for every hit, which is much slower than using doc values.

Try this instead

```auto
return doc['log.keyword'].size() > 0 && doc['log.keyword'].value.length() > 100

```

--Alex

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 19, 2020, 1:56pm UTC](https://discuss.elastic.co/t/how-to-use-source-to-create-a-scripted-filter/219646/13 "2020-03-19T13:56:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.