Elasticsearch Version
Version: 8.11.1, Build: rpm/6f9ff581fbcde658e6f69d6ce03050f060d1fd0c/2023-11-11T10:05:59.421038163Z, JVM: 21.0.1
Installed Plugins
Java Version
openjdk version "21.0.1"
OS Version
6.1.61-85.141.amzn2023.x86_64
Problem Description
I often use map_script for statistics, but when I use Debug.explain, the data nodes emit the message below and all die.
If removing Debug.explain, it works normally.
It wasn't a problem when using version 8.6.2, but since upgrading, I'm seeing that.
I need help.
[ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [data-03] fatal error in thread [elasticsearch[data-03][search_worker][T#14]], exiting
org.elasticsearch.painless.PainlessExplainError: null
at org.elasticsearch.painless.api.Debug.explain(Debug.java:23) ~[?:?]
at org.elasticsearch.painless.PainlessScript$Script.execute( ...:66) ~[?:?]
at org.elasticsearch.search.aggregations.metrics.ScriptedMetricAggregator$1.collect(ScriptedMetricAggregator.java:124) ~[elasticsearch-8.11.1.jar:?]
at org.elasticsearch.search.aggregations.LeafBucketCollector.collect(LeafBucketCollector.java:86) ~[elasticsearch-8.11.1.jar:?]
at org.elasticsearch.search.query.QueryPhaseCollector$CompositeLeafCollector.collect(QueryPhaseCollector.java:306) ~[elasticsearch-8.11.1.jar:?]
at org.apache.lucene.search.Weight$DefaultBulkScorer.scoreRange(Weight.java:277) ~[lucene-core-9.8.0.jar:?]
at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:236) ~[lucene-core-9.8.0.jar:?]
at org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:45) ~[elasticsearch-8.11.1.jar:?]
at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:38) ~[lucene-core-9.8.0.jar:?]
at org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:536) ~[elasticsearch-8.11.1.jar:?]
at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:460) ~[elasticsearch-8.11.1.jar:?]
at org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:375) ~[elasticsearch-8.11.1.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.11.1.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.11.1.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.11.1.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Steps to Reproduce
- create index
PUT /test_index?pretty
{
"settings" : {
"number_of_shards" : 1,
"number_of_replicas" : 0
},
"mappings" : {
"properties" : {
"tags" : { "type" : "keyword" },
"updated_at" : { "type" : "date" }
}
}
}
- index a document
PUT test_index/_doc/1
{
"tags": [
"opster",
"elasticsearch"
],
"date": "01-01-2020"
}
- search
GET test_index/_search
{
"aggs": {
"test":{
"scripted_metric":{
"init_script": """
state.funding = new HashMap();
""",
"map_script": """
def source = params._source;
Debug.explain(source);
return;
""",
"combine_script":"""
return state;
""",
"reduce_script": """
def result = new HashMap();
return result;
"""
}
}
},
"size":0
}
- check
GET test_index/_search
If you use Debug.explain in map_script, an error message will appear, but if you search another index afterwards, the search will not work. The reason is that the data nodes are dead with the below message.
Logs (if relevant)
- map_script result
{
"error": {
"root_cause": [
{
"type": "script_exception",
"reason": "runtime error",
"painless_class": "java.util.LinkedHashMap",
"to_string": "{tags=[opster, elasticsearch], date=01-01-2020}",
"java_class": "java.util.LinkedHashMap",
"script_stack": [
"""Debug.explain(source);
""",
" ^---- HERE"
],
"script": " ...",
"lang": "painless",
"position": {
"offset": 65,
"start": 51,
"end": 84
}
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "test_index",
"node": "o39edlW3TC6VvO9Iu0bH8w",
"reason": {
"type": "script_exception",
"reason": "runtime error",
"painless_class": "java.util.LinkedHashMap",
"to_string": "{tags=[opster, elasticsearch], date=01-01-2020}",
"java_class": "java.util.LinkedHashMap",
"script_stack": [
"""Debug.explain(source);
""",
" ^---- HERE"
],
"script": " ...",
"lang": "painless",
"position": {
"offset": 65,
"start": 51,
"end": 84
},
"caused_by": {
"type": "painless_explain_error",
"reason": null
}
}
}
]
},
"status": 400
}
- elasticsearch log
[ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [master-1] fatal error in thread [elasticsearch[master-1][search_worker][T#5]], exiting
org.elasticsearch.painless.PainlessExplainError: null
at org.elasticsearch.painless.api.Debug.explain(Debug.java:23) ~[?:?]
at org.elasticsearch.painless.PainlessScript$Script.execute( ...:66) ~[?:?]
at org.elasticsearch.search.aggregations.metrics.ScriptedMetricAggregator$1.collect(ScriptedMetricAggregator.java:124) ~[elasticsearch-8.11.1.jar:?]
at org.elasticsearch.search.aggregations.LeafBucketCollector.collect(LeafBucketCollector.java:86) ~[elasticsearch-8.11.1.jar:?]
at org.apache.lucene.search.MatchAllDocsQuery$1$1.score(MatchAllDocsQuery.java:63) ~[lucene-core-9.8.0.jar:?]
at org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:45) ~[elasticsearch-8.11.1.jar:?]
at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:38) ~[lucene-core-9.8.0.jar:?]
at org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:536) ~[elasticsearch-8.11.1.jar:?]
at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:460) ~[elasticsearch-8.11.1.jar:?]
at org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:375) ~[elasticsearch-8.11.1.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.11.1.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.11.1.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.11.1.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1583) ~[?:?]
- system log
systemd-entrypoint: ERROR: Elasticsearch exited unexpectedly, with exit code 1