Hi I am trying to write a query to return documents which has the length of the log field more than 900. It is a text field.
Running the below search I get variable _source is not defined
GET logdata/_search
{
"query": {
"bool": {
"filter": {
"script": {
"script": {
"source": "_source['log'].length() > 900",
"lang": "painless"
}
}
}
}
}
}
Any help appreciated.
Thanks
try something like doc['log.keyword'].value.length() (untested, on top of my head).
Thanks for the reply. No, it doesn't work. I get the same error.
Tried in ver 6.7
If you do not provide error messages it will be impossible to help further.
Please provide a fully reproducible, but minimal example including index creation with mappings, two sample documents (consisting only of that single field you are using in your query) being indexed and the query you are using, plus responses if they contain an error.
Thanks
Hi,
Apologies, I thought it should be same for any text field. I have provided the details below.
I use the exact same query posted in the question
Document
{
"deviceid" : "XX12",
"time" : "2020-02-20T00:00:00.000Z",
"severity" : "notice",
"log" : "[system current time] : Thu Feb 20 00:00:00 UTC 2020",
}
Mapping
{
"logdata" : {
"mappings" : {
"loginfo" : {
"properties" : {
"deviceid" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"log" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"severity" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"time" : {
"type" : "date"
}
}
}
}
}
}
Error message
{
"error": {
"root_cause": [
{
"type": "script_exception",
"reason": "compile error",
"script_stack": [
"_source['log'].va ...",
"^---- HERE"
],
"script": "_source['log'].value.length() > 10",
"lang": "painless"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "logdata",
"node": "xxxxxxxxxxxxxx",
"reason": {
"type": "query_shard_exception",
"reason": "failed to create query: {\n "bool" : {\n "filter" : [\n {\n "script" : {\n "script" : {\n "source" : "_source['log'].value.length() > 10",\n "lang" : "painless"\n },\n "boost" : 1.0\n }\n }\n ],\n "adjust_pure_negative" : true,\n "boost" : 1.0\n }\n}",
"index_uuid": "index_uuid",
"index": "logdata",
"caused_by": {
"type": "script_exception",
"reason": "compile error",
"script_stack": [
"_source['log'].va ...",
"^---- HERE"
],
"script": "_source['log'].value.length() > 10",
"lang": "painless",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Variable [_source] is not defined."
}
}
}
}
],
"caused_by": {
"type": "script_exception",
"reason": "compile error",
"script_stack": [
"_source['log'].va ...",
"^---- HERE"
],
"script": "_source['log'].value.length() > 10",
"lang": "painless",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Variable [_source] is not defined."
}
}
},
"status": 400
}
your sample does not show my suggestion but still yours which returns an error.
Please find the error messages for your suggestion below.
With keyword
{
"error": {
"root_cause": [
{
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"doc['log.keyword'].value.length() > 10",
" ^---- HERE"
],
"script": "doc['log.keyword'].value.length() > 10",
"lang": "painless"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "logdata",
"node": "xxxxxxxxxxxxx",
"reason": {
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"doc['log.keyword'].value.length() > 10",
" ^---- HERE"
],
"script": "doc['log.keyword'].value.length() > 10",
"lang": "painless",
"caused_by": {
"type": "null_pointer_exception",
"reason": null
}
}
}
]
},
"status": 500
}
Without keyword
{
"error": {
"root_cause": [
{
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:779)",
"org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",
"org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:283)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:88)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:85)",
"java.security.AccessController.doPrivileged(Native Method)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:85)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:39)",
"doc['log'].value.length() > 10",
" ^---- HERE"
],
"script": "doc['log'].value.length() > 10",
"lang": "painless"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "logdata",
"node": "xxxxxxxxxxxx",
"reason": {
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:779)",
"org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",
"org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:283)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:88)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:85)",
"java.security.AccessController.doPrivileged(Native Method)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:85)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:39)",
"doc['log'].value.length() > 10",
" ^---- HERE"
],
"script": "doc['log'].value.length() > 10",
"lang": "painless",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [log] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
}
}
}
]
},
"status": 500
}
is it possible that not all of your documents have the log field set?
Not possible. I checked again using exists query all documents have log field.
now helping is hard without concrete data. Can you create a minimal but reproducible example that includes index creation, mapping, two sample documents so that one can reproduce that issue?
Thanks!
Hi @spinscale
I have shared the document, search and mapping.
log.keyword doesn't work because it is not available in all documents. Like I said in my earlier replies it has ignore_above:256 in the mapping
So I created an test index with only one document.
Then, this works
But what I am interested is the log text field which is present in all the documents.
doc['log'].value.length doesn't work.
{
"error": {
"root_cause": [
{
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:779)",
"org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",
"org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:283)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:88)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:85)",
"java.security.AccessController.doPrivileged(Native Method)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:85)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:39)",
"doc['log'].value.length() > 4",
" ^---- HERE"
],
"script": "doc['log'].value.length() > 4",
"lang": "painless"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "logdata",
"node": "E486K0eoRnyCBTJv7WBTrg",
"reason": {
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:779)",
"org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",
"org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:283)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:88)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:85)",
"java.security.AccessController.doPrivileged(Native Method)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:85)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:39)",
"doc['log'].value.length() > 4",
" ^---- HERE"
],
"script": "doc['log'].value.length() > 4",
"lang": "painless",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [log] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
}
}
}
]
},
"status": 500
}
What I am interested is to use the _source field to make this work. Is there any way to do this?
Hey,
so the _source field cannot be used, as this would mean, that the JSON source needs to be extracted for every hit, which is much slower than using doc values.
Try this instead
return doc['log.keyword'].size() > 0 && doc['log.keyword'].value.length() > 100
--Alex
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.