Elasticsearch gives execption while report download["message":"Request cannot be executed; I/O reactor status: STOPPED","errorstacktrace":"java.lang.RuntimeException]

Hello All,

I'm getting following exception while downloading report by query elastic.Unable to understand resolution for this.Tried increasing the jvm heap to 12 gb ,still same exception.

Exception:

{"statusCode":"REPORT_SERVICE_1009","timestamp":"2023-01-10T11:23:54.568+00:00","message":"Request cannot be executed; I/O reactor status: STOPPED","errorstacktrace":"java.lang.RuntimeException: Request cannot be executed; I/O reactor status: STOPPED\n\tat org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:889)\n\tat org.elasticsearch.client.RestClient.performRequest(RestClient.java:283)\n\tat org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)\n\tat org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2082)\n\tat org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1732)\n\tat org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1702)\n\tat org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1672)\n\tat org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1128)\n\tat new.connect.integration.services.reportservice.service.randReportService.getrandUniqueIds(randReportService.java:282)\n\tat new.connect.integration.services.reportservice.service.randReportService.retrieveImportJobsData(randReportService.java:128)\n\tat new.connect.integration.services.reportservice.service.randReportService.generaterandReport(randReportService.java:96)\n\tat new.connect.integration.services.reportservice.controller.ReportServiceController.downloadrandReport(ReportServiceController.java:1045)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566)\n\tat org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)\n\tat org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)\n\tat org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)\n\tat org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)\n\tat org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)\n\tat org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)\n\tat org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)\n\tat org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:655)\n\tat org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:764)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)\n\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)\n\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)\n\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)\n\tat org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)\n\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)\n\tat org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)\n\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)\n\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889)\n\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743)\n\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)\n\tat org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)\n\tat org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\nCaused by: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED\n\tat org.apache.http.util.Asserts.check(Asserts.java:46)\n\tat org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)\n\tat org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)\n\tat org.elasticsearch.client.RestClient.performRequest(RestClient.java:279)\n\t... 64 more\n"}

Query:

GET /his-monitor-adapter-*/_search
{
  "query": {
 GET /tis-monitor-adapter-*/_search
{
  "size": 0,
  "query": {
    "bool": {
      "must": [
        {
          "exists": {
            "field": "adapter.scheduler"
          }
        }
      ]
    }
  }, 
  "aggs": {
    "uniqueIdList": {
      "terms": {
        "field": "adapter.uniqueId.keyword",
        "size": 10000
      }
    }
  }
}

TEST INSTANCE:

How many indices and shards does it run against?
1 Index,1 shard

How much data do those indices contain?
8840 documents,1.4mb

What is the specification of your Elasticsearch cluster?
Single Node elastic cluster,8 core CPU

JVM HEAP Allocated:
-Xms8g
-Xmx8g

Any suggestion would be helpful.

Thanx

Have a look at: Low-level REST client status error:ava.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED · Issue #42133 · elastic/elasticsearch (github.com)
and
Elasticsearch Request cannot be executed; I/O reactor status: STOPPED - Stack Overflow
It's worth checking on how many connections are being created from your client to ES, creating too many connections or not closing them properly can lead to this situation.

Hello @Ayush_Mathur ,

Thanx for your time for looking into this and for links mentioned.Will look into points you've mentioned.

Many Thanx

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.