I am running into an issue that I believe has to do with the ES Java API client. I can make the following request to my ES cluster in Postman and receive the intended results:
GET resumes/_search?pretty=true
{
"size": 0,
"runtime_mappings" : {
"name_and_email": {
"type" : "keyword",
"script" : { "source" : "emit(doc['lastname.keyword'].value + doc['firstname.keyword'].value)"}
}
},
"aggs" : {
"unique_resumes" : {
"terms" : {
"field" : "name_and_email"
}
}
}
}
Which yields a response of:
{
"took": 15,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1653,
"relation": "eq"
},
"max_score": null,
"hits": []
},
"aggregations": {
"unique_resumes": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 1602,
"buckets": [
{
"key": "SwiftTaylor",
"doc_count": 11
},
{
"key": "PittBrad",
"doc_count": 6
},
{
"key": "BradyTom",
"doc_count": 5
},
{
"key": "MossRandy",
"doc_count": 5
},
{
"key": "WilsonRussel",
"doc_count": 5
},
{
"key": "BushGeorge",
"doc_count": 4
},
{
"key": "ObamaBarrack",
"doc_count": 4
},
{
"key": "SmithJohn",
"doc_count": 4
},
{
"key": "DoeJohn",
"doc_count": 4
},
{
"key": "DoeJane",
"doc_count": 3
}
]
}
}
}
I am admittedly unfamiliar with aggregates in ES so I am not certain that this is the perfect way to do it, but for now it works in postman. The problem arises when I implement that query using the Java API Client in my spring boot application. I tried using a hardcoded string and the .withJson() method to ensure that the query is being translated perfectly like this:
private final String jsonQuery = " { \r\n"
+ " \"size\": 0,\r\n"
+ " \"runtime_mappings\" : {\r\n"
+ " \"name_and_email\": {\r\n"
+ " \"type\" : \"keyword\",\r\n"
+ " \"script\" : { \"source\" : \"emit(doc['firstname.keyword'].value + doc['lastname.keyword'].value)\"}\r\n"
+ " }\r\n"
+ " },\r\n"
+ " \"aggs\" : {\r\n"
+ " \"unique_resumes\" : {\r\n"
+ " \"terms\" : {\r\n"
+ " \"field\" : \"name_and_email\"\r\n"
+ " }\r\n"
+ " }\r\n"
+ " }\r\n"
+ "\r\n"
+ "} ";
SearchResponse<JsonNode> totalQuery = es.getClient().search(s->s
.withJson(new ByteArrayInputStream(jsonQuery.getBytes())), JsonNode.class);
This returns an error like such from the client:
co.elastic.clients.elasticsearch._types.ElasticsearchException: [es/search] failed: [search_phase_execution_exception] all shards failed
at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:281)
at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:147)
...
So I looked in my ES logs and found this error:
[2022-07-20T16:01:44,659][TRACE][o.e.s.SearchService ] [rdb-elasticsearch-8-dev-node] Query phase failed
org.elasticsearch.index.mapper.MapperParsingException: Expected map for runtime field [name_and_email] definition but got a java.util.ArrayList
at org.elasticsearch.index.mapper.RuntimeField.parseRuntimeFields(RuntimeField.java:188) ~[elasticsearch-8.3.0.jar:?]
at org.elasticsearch.index.mapper.RuntimeField.parseRuntimeFields(RuntimeField.java:132) ~[elasticsearch-8.3.0.jar:?]
at org.elasticsearch.index.query.SearchExecutionContext.parseRuntimeMappings(SearchExecutionContext.java:685) ~[elasticsearch-8.3.0.jar:?]
at org.elasticsearch.index.query.SearchExecutionContext.<init>(SearchExecutionContext.java:162) ~[elasticsearch-8.3.0.jar:?]
at org.elasticsearch.index.IndexService.newSearchExecutionContext(IndexService.java:631) ~[elasticsearch-8.3.0.jar:?]
at org.elasticsearch.search.DefaultSearchContext.<init>(DefaultSearchContext.java:163) ~[elasticsearch-8.3.0.jar:?]
at org.elasticsearch.search.SearchService.createSearchContext(SearchService.java:1020) ~[elasticsearch-8.3.0.jar:?]
at org.elasticsearch.search.SearchService.createContext(SearchService.java:968) ~[elasticsearch-8.3.0.jar:?]
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:623) [elasticsearch-8.3.0.jar:?]
at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:489) [elasticsearch-8.3.0.jar:?]
at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) [elasticsearch-8.3.0.jar:?]
at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) [elasticsearch-8.3.0.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.3.0.jar:?]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) [elasticsearch-8.3.0.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:768) [elasticsearch-8.3.0.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.3.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
I used the correct/preferred way using the lambda syntax and received the exact same error:
SearchResponse<JsonNode> totalQuery = es.getClient().search(s->s
.size(0)
.runtimeMappings("name_and_email", Arrays.asList(
RuntimeField.of(r->r
.type(RuntimeFieldType.Keyword)
.script(p->p
.inline(i->i
.lang("painless")
.source("emit(doc['firstname.keyword'].value + doc['lastname.keyword'].value )")
)
)
)
)
)
.aggregations("unique_resumes", Aggregation.of(a->a
.terms(t->t
.field("name_and_email")
)
)
)
, JsonNode.class);
To the best of my knowledge this is a 1 to 1 mapping of the query I am trying to perform. One bit of confusion I have, is that when adding runtimeMappings, it wants a key with a list of runtime mappings so I use Arrays.asList to do so. If it were not for the first method failing as well, I would have thought that the error originated there but it appears that the string is interpreted in the same way.
The error makes sense with the lambda expression way, as I am passing an array list in with the key "name_and_email". However, the same error is thrown when I am just passing the query as a json string.
Any help or suggestions would be appreciated.