Repeated NoNodeAvailableExceptions Exceptions

Hi all. I'm finding that I am getting repeated NodeDisconnectedException and NoNodeAvailableExceptions exceptions when running relatively few concurrent queries - fewer than 20.

I think I am beginning to understand the reason why but some confirmation would be most appreciated.

The default queue size is 1000 and the default number of shards per index is 5. A query against each shard uses 1 in the queue. So, for example, querying against 20 indexes with 5 shards each you would only need to run 101 concurrent queries to go over the default queue size and cause these exceptions?

Is my understanding correct?

Thanks,

Ben.

That queue size is for the action-specific threadpool, it should not affect the
NodeDisconnectedException and NoNodeAvailableException failures that you are
seeing, those exceptions are because either a node has been disconnected and
detected with fault detection, or a node cannot be contacted.

You should check the logs for the full message for the node disconnection
exceptions and see whether they failed during communication.

Hi Lee,

Thanks for the reply. I am not seeing anything in the Elasticsearch logs on either node. All the errors are logged on the client side which I have pasted below.

I have tried a few options on the Transport client such as turning sniff on and increasing the ping and node sampler interval values but this has made no difference. Is there anything I can do to help debug the error?

I am using v5.1.2 of both Elasticsearch and the Java client. My setup consists of two virtual machines each with 2 CPUs, 4GB RAM and 2GB allocated to the JVM.

Thanks.

None of the configured nodes were available: [{escluster-es2}{-iqECyceSn2FBSPbg-nFvw}{F2PCjdZTTUujzj5996um6w}{192.168.0.222}{192.168.0.222:9300}, {escluster-es1}{WELN6zJ8QYy40zew1Axs2w}{Zc4W94tbT7exlRHziAnd-g}{192.168.0.221}{192.168.0.221:9300}]: NoNodeAvailableException[None of the configured nodes were available: [{escluster-es2}{-iqECyceSn2FBSPbg-nFvw}{F2PCjdZTTUujzj5996um6w}{192.168.0.222}{192.168.0.222:9300}, {escluster-es1}{WELN6zJ8QYy40zew1Axs2w}{Zc4W94tbT7exlRHziAnd-g}{192.168.0.221}{192.168.0.221:9300}]]; nested: NodeDisconnectedException[[escluster-es1][192.168.0.221:9300][indices:data/read/search] disconnected];
at org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:266)
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51)
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:955)
at org.elasticsearch.transport.TransportService$Adapter.lambda$raiseNodeDisconnected$3(TransportService.java:783)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:451)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: NodeDisconnectedException[[escluster-es1][192.168.0.221:9300][indices:data/read/search] disconnected]

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.