Search.max_async_search_response_size

Hi all, I am new to Elasticsearch stack. Any suggestions or help is greatly appreciated please.

I encountered this error after doing a http_poller in Logstash.

{"@timestamp":"2022-10-21T13:17:56.029Z", "log.level":"ERROR", "message":"failed to store async-search [1PDSAmmPTa2gwQ2kO422zA]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[f51a793c452f][search][T#4]","log.logger":"org.elasticsearch.xpack.core.async.AsyncTaskIndexService","trace.id":"7564221ed4d1baa1526b98f877c18a1c","elasticsearch.cluster.uuid":"2eLAL11BRxKy_Bp9dBV1Ew","elasticsearch.node.id":"8c4HrEMLR3yWy9FmPmCKXQ","elasticsearch.node.name":"f51a793c452f","elasticsearch.cluster.name":"docker-cluster","error.type":"java.lang.IllegalArgumentException","error.message":"Can't store an async search response larger than [10485760] bytes. This limit can be set by changing the [search.max_async_search_response_size] setting.","error.stack_trace":"java.lang.IllegalArgumentException: Can't store an async search response larger than [10485760] bytes. This limit can be set by changing the [search.max_async_search_response_size] setting.\n\tat org.elasticsearch.xcore@8.4.1/org.elasticsearch.xpack.core.async.AsyncTaskIndexService$ReleasableBytesStreamOutputWithLimit.ensureCapacity(AsyncTaskIndexService.java:634)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.stream.BytesStreamOutput.writeBytes(BytesStreamOutput.java:86)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.stream.StreamOutput.write(StreamOutput.java:504)\n\tat org.elasticsearch.base@8.4.1/org.elasticsearch.core.Streams$1.write(Streams.java:92)\n\tat java.base/java.util.Base64$EncOutputStream.write(Base64.java:973)\n\tat org.elasticsearch.base@8.4.1/org.elasticsearch.core.Streams$1.write(Streams.java:92)\n\tat java.base/java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:261)\n\tat java.base/java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:210)\n\tat java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)\n\tat java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:127)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.stream.OutputStreamStreamOutput.writeBytes(OutputStreamStreamOutput.java:29)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.stream.StreamOutput.writeBytes(StreamOutput.java:121)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.stream.StreamOutput.writeString(StreamOutput.java:433)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.stream.StreamOutput.writeGenericString(StreamOutput.java:782)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.stream.StreamOutput.lambda$static$6(StreamOutput.java:649)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.stream.StreamOutput.writeGenericValue(StreamOutput.java:820)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.stream.StreamOutput.writeCollection(StreamOutput.java:1160)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.document.DocumentField.writeTo(DocumentField.java:118)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.search.SearchHit.lambda$writeTo$1(SearchHit.java:259)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.stream.StreamOutput.writeMap(StreamOutput.java:624)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.search.SearchHit.writeTo(SearchHit.java:259)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.stream.StreamOutput.lambda$writeArray$31(StreamOutput.java:939)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.stream.StreamOutput.writeArray(StreamOutput.java:916)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.stream.StreamOutput.writeArray(StreamOutput.java:939)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.search.SearchHits.writeTo(SearchHits.java:100)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.search.internal.InternalSearchResponse.writeTo(InternalSearchResponse.java:73)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.action.search.SearchResponse.writeTo(SearchResponse.java:434)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.stream.StreamOutput.writeOptionalWriteable(StreamOutput.java:953)\n\tat org.elasticsearch.xcore@8.4.1/org.elasticsearch.xpack.core.search.action.AsyncSearchResponse.writeTo(AsyncSearchResponse.java:96)\n\tat org.elasticsearch.xcore@8.4.1/org.elasticsearch.xpack.core.async.AsyncTaskIndexService.writeResponse(AsyncTaskIndexService.java:578)\n\tat org.elasticsearch.xcore@8.4.1/org.elasticsearch.xpack.core.async.AsyncTaskIndexService.lambda$updateResponse$3(AsyncTaskIndexService.java:291)\n\tat org.elasticsearch.xcontent@8.4.1/org.elasticsearch.xcontent.XContentBuilder.lambda$directFieldAsBase64$24(XContentBuilder.java:1212)\n\tat org.elasticsearch.xcontent.impl@8.4.1/org.elasticsearch.xcontent.provider.json.JsonXContentGenerator.writeDirectField(JsonXContentGenerator.java:557)\n\tat org.elasticsearch.xcontent@8.4.1/org.elasticsearch.xcontent.XContentBuilder.directFieldAsBase64(XContentBuilder.java:1206)\n\tat org.elasticsearch.xcore@8.4.1/org.elasticsearch.xpack.core.async.AsyncTaskIndexService.updateResponse(AsyncTaskIndexService.java:291)\n\tat org.elasticsearch.xcore@8.4.1/org.elasticsearch.xpack.core.async.AsyncTaskIndexService.updateResponse(AsyncTaskIndexService.java:270)\n\tat org.elasticsearch.xpack.search.TransportSubmitAsyncSearchAction.onFinalResponse(TransportSubmitAsyncSearchAction.java:204)\n\tat org.elasticsearch.xpack.search.TransportSubmitAsyncSearchAction$1$1.lambda$onResponse$1(TransportSubmitAsyncSearchAction.java:106)\n\tat org.elasticsearch.xpack.search.AsyncSearchTask.executeCompletionListeners(AsyncSearchTask.java:307)\n\tat org.elasticsearch.xpack.search.AsyncSearchTask$Listener.onResponse(AsyncSearchTask.java:446)\n\tat org.elasticsearch.xpack.search.AsyncSearchTask$Listener.onResponse(AsyncSearchTask.java:367)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.security@8.4.1/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:165)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:245)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.action.ActionListener$RunAfterActionListener.onResponse(ActionListener.java:367)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.sendSearchResponse(AbstractSearchAsyncAction.java:722)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.action.search.FetchLookupFieldsPhase.run(FetchLookupFieldsPhase.java:75)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:469)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:463)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.action.search.ExpandSearchPhase.onPhaseDone(ExpandSearchPhase.java:151)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:105)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:469)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:463)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:271)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.action.search.FetchSearchPhase.lambda$innerRun$2(FetchSearchPhase.java:108)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:117)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:90)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:769)\n\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n\tSuppressed: java.lang.IllegalArgumentException: Can't store an async search response larger than [10485760] bytes. This limit can be set by changing the [search.max_async_search_response_size] setting.\n\t\tat org.elasticsearch.xcore@8.4.1/org.elasticsearch.xpack.core.async.AsyncTaskIndexService$ReleasableBytesStreamOutputWithLimit.ensureCapacity(AsyncTaskIndexService.java:634)\n\t\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.stream.BytesStreamOutput.writeBytes(BytesStreamOutput.java:86)\n\t\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.stream.StreamOutput.write(StreamOutput.java:504)\n\t\tat org.elasticsearch.base@8.4.1/org.elasticsearch.core.Streams$1.write(Streams.java:92)\n\t\tat java.base/java.util.Base64$EncOutputStream.write(Base64.java:973)\n\t\tat org.elasticsearch.base@8.4.1/org.elasticsearch.core.Streams$1.write(Streams.java:92)\n\t\tat java.base/java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:261)\n\t\tat java.base/java.util.zip.DeflaterOutputStream.finish(DeflaterOutputStream.java:226)\n\t\tat java.base/java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:244)\n\t\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.compress.DeflateCompressor$2.close(DeflateCompressor.java:186)\n\t\tat java.base/java.io.FilterOutputStream.close(FilterOutputStream.java:191)\n\t\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.stream.OutputStreamStreamOutput.close(OutputStreamStreamOutput.java:39)\n\t\tat org.elasticsearch.xcore@8.4.1/org.elasticsearch.xpack.core.async.AsyncTaskIndexService.writeResponse(AsyncTaskIndexService.java:576)\n\t\t... 34 more\n\t\tSuppressed: java.lang.IllegalArgumentException: Can't store an async search response larger than [10485760] bytes. This limit can be set by changing the [search.max_async_search_response_size] setting.\n\t\t\tat org.elasticsearch.xcore@8.4.1/org.elasticsearch.xpack.core.async.AsyncTaskIndexService$ReleasableBytesStreamOutputWithLimit.ensureCapacity(AsyncTaskIndexService.java:634)\n\t\t\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.stream.BytesStreamOutput.writeBytes(BytesStreamOutput.java:86)\n\t\t\tat org.elasticsearch.server@8.4.1/org.elasticsearch.common.io.stream.StreamOutput.write(StreamOutput.java:504)\n\t\t\tat org.elasticsearch.base@8.4.1/org.elasticsearch.core.Streams$1.write(Streams.java:92)\n\t\t\tat java.base/java.util.Base64$EncOutputStream.write(Base64.java:973)\n\t\t\tat org.elasticsearch.base@8.4.1/org.elasticsearch.core.Streams$1.write(Streams.java:92)\n\t\t\tat java.base/java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:261)\n\t\t\tat java.base/java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:210)\n\t\t\tat java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)\n\t\t\tat java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)\n\t\t\tat java.base/java.io.FilterOutputStream.close(FilterOutputStream.java:182)\n\t\t\t... 36 more\n"}
{"@timestamp":"2022-10-21T13:17:59.055Z", "log.level": "INFO", "message":"[.kibana_8.4.1_001/q9MAEh_KSFeF_NuPIrl7NQ] update_mapping [_doc]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[f51a793c452f][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataMappingService","trace.id":"7564221ed4d1baa1526b98f877c18a1c","elasticsearch.cluster.uuid":"2eLAL11BRxKy_Bp9dBV1Ew","elasticsearch.node.id":"8c4HrEMLR3yWy9FmPmCKXQ","elasticsearch.node.name":"f51a793c452f","elasticsearch.cluster.name":"docker-cluster"}

ERROR: Elasticsearch exited unexpectedly

It should be related to Can't store an async search response larger than [10485760] bytes. This limit can be set by changing the [search.max_async_search_response_size] setting.

Anyone know the max size i can set for this?

Currently, i am testing for search.max_async_search_response_size: 1048576000 in elasticsearch.yml

*finger Crossed

I have removed this code in yml after research and it works better
id => "pa_http"

Faced with this error:
The content length (612290936) is bigger than the maximum allowed string (536870888)

handleSearchError@http://localhost:5601/55434/bundles/plugin/data/kibana/data.plugin.js:1:409692
search/</<@http://localhost:5601/55434/bundles/plugin/data/kibana/data.plugin.js:1:412433
a/</s<@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:364:106267
t/s._error<@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:6:29525
__kbnSharedDeps_npm__</f</t.prototype.error@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:334:88987
__kbnSharedDeps_npm__</c</t.prototype.error/<@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:329:46099
o@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:334:252982
__kbnSharedDeps_npm__</c</t.prototype.error@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:329:45966
error@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:334:430606
__kbnSharedDeps_npm__</m</e.prototype.error@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:334:89793
__kbnSharedDeps_npm__</f</t.prototype._error@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:334:89350
__kbnSharedDeps_npm__</f</t.prototype.error@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:334:88987
o@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:364:254482
__kbnSharedDeps_npm__</l</e.prototype._trySubscribe@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:272:15794
__kbnSharedDeps_npm__</l</e.prototype.subscribe/<@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:272:15716
o@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:334:252982
__kbnSharedDeps_npm__</l</e.prototype.subscribe@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:272:15631
a/</s<@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:364:106309
t/s._error<@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:6:29525
__kbnSharedDeps_npm__</f</t.prototype.error@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:334:88987
s/</<@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:364:125124
t/s._error<@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:6:29525
__kbnSharedDeps_npm__</f</t.prototype.error@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:334:88987
__kbnSharedDeps_npm__</f</t.prototype._error@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:334:89350
__kbnSharedDeps_npm__</f</t.prototype.error@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:334:88987
s/</<@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:364:125124
t/s._error<@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:6:29525
__kbnSharedDeps_npm__</f</t.prototype.error@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:334:88987
__kbnSharedDeps_npm__</f</t.prototype._error@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:334:89350
__kbnSharedDeps_npm__</f</t.prototype.error@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:334:88987
__kbnSharedDeps_npm__</f</t.prototype._error@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:334:89350
__kbnSharedDeps_npm__</f</t.prototype.error@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:334:88987
v/</<@http://localhost:5601/55434/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:272:7745

This edit in kibana.yml seems to work well

server.maxPayload: 553247624
savedObjects.maxImportPayloadBytes: 553247624

Faced with another async error:

{"@timestamp":"2022-10-22T14:38:39.157Z", "log.level": "WARN", "message":"handling request [fafbdfb8-d824-413c-93e0-94b9a2d78b20;kibana:application:discover:new;application:discover:new][GET][/_async_search/FlNQNVdqb2RFUm9LZk5JQnVYVGtOZGcbYndMdkExRDZSMkdobFV1ckpXV0RDUTo1OTEw?wait_for_completion_timeout=100ms][Netty4HttpChannel{localAddress=/172.23.0.2:9200, remoteAddress=/172.23.0.4:33824}] took [6859ms] which is above the warn threshold of [5000ms]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[eefc7b01d62a][transport_worker][T#3]","log.logger":"org.elasticsearch.http.AbstractHttpServerTransport","elasticsearch.cluster.uuid":"VVXCfmDJSBSDhHrogaezKA","elasticsearch.node.id":"bwLvA1D6R2GhlUurJWWDCQ","elasticsearch.node.name":"eefc7b01d62a","elasticsearch.cluster.name":"docker-cluster"}

ERROR: Elasticsearch exited unexpectedly

Anyone knows how to mitigate?

How much JVM memory are you giving elasticsearch?

ES_JAVA_OPTS: -Xms2g -Xmx2g

i tried to remove redundant fields in logstash and it seems alot better.
My macbook is 8GB as I am running POC first on it.

Don't forget you also need to set the resource setting in the docker desktop to provide enough resources for everything you are running in docker.

Thanks! I didn't know about this!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.