Bad query causes all processing to stop temporarily

Elastic Stack Elasticsearch
1

I recently ran into an issue where if a pretty bad query was executed it caused all processing to stop for a couple minutes. By all processing I mean all searches timed out and I could not push any docs to elasticsearch.

Here is the query: "http://10.0.2.1:9200/-/_search"

The error generated in the elasticsearch log is below, I've fixed the bad query, but my question is how can I avoid this so that when a bad query is executed it does not shut down all elasticsearch processing ?

org.elasticsearch.transport.RemoteTransportException: [rEDv2RX][10.0.2.1:9300][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.transport.TransportService$7@36aed76b on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@531cbf14[Running, pool size = 5, active threads = 5, queued tasks = 1000, completed tasks = 27593]]
        at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:50) ~[elasticsearch-5.3.1.jar:5.3.1]
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) ~[?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) ~[?:1.8.0_131]
        at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.doExecute(EsThreadPoolExecutor.java:94) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:89) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:614) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.transport.TransportService.access$000(TransportService.java:73) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:133) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:562) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:495) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:522) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:513) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:138) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.sendExecuteFirstPhase(SearchQueryThenFetchAsyncAction.java:58) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.performInitialPhase(AbstractSearchAsyncAction.java:140) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:119) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:240) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:146) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:67) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:404) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:531) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.rest.action.search.RestSearchAction.lambda$prepareRequest$1(RestSearchAction.java:73) ~[elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:80) [elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:260) [elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:199) [elasticsearch-5.3.1.jar:5.3.1]
        at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:505) [transport-netty4-5.3.1.jar:5.3.1]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:72) [transport-netty4-5.3.1.jar:5.3.1]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
        at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:63) [transport-netty4-5.3.1.jar:5.3.1]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
        at
2

This sort of thing is where we are doing more work to limit and ultimately stop such bad queries. For the moment you would be best of putting some kind of access control in place.

3

There is a timeout parameter you can set which can help shut down long-running queries.
It isn't guaranteed to curtail all-long running queries but the major "hot-loops" in query excecution have added code to check that the search is not over-running.

1 Like
4

Thanks guys I appreciate it.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.