Repeated NoNodeAvailableExceptions Exceptions

Elastic Stack Elasticsearch
1

Hi all. I'm finding that I am getting repeated NodeDisconnectedException and NoNodeAvailableExceptions exceptions when running relatively few concurrent queries - fewer than 20.

I think I am beginning to understand the reason why but some confirmation would be most appreciated.

The default queue size is 1000 and the default number of shards per index is 5. A query against each shard uses 1 in the queue. So, for example, querying against 20 indexes with 5 shards each you would only need to run 101 concurrent queries to go over the default queue size and cause these exceptions?

Is my understanding correct?

Thanks,

Ben.

2

That queue size is for the action-specific threadpool, it should not affect the
NodeDisconnectedException and NoNodeAvailableException failures that you are
seeing, those exceptions are because either a node has been disconnected and
detected with fault detection, or a node cannot be contacted.

You should check the logs for the full message for the node disconnection
exceptions and see whether they failed during communication.

3

Hi Lee,

Thanks for the reply. I am not seeing anything in the Elasticsearch logs on either node. All the errors are logged on the client side which I have pasted below.

I have tried a few options on the Transport client such as turning sniff on and increasing the ping and node sampler interval values but this has made no difference. Is there anything I can do to help debug the error?

I am using v5.1.2 of both Elasticsearch and the Java client. My setup consists of two virtual machines each with 2 CPUs, 4GB RAM and 2GB allocated to the JVM.

Thanks.

None of the configured nodes were available: [{escluster-es2}{-iqECyceSn2FBSPbg-nFvw}{F2PCjdZTTUujzj5996um6w}{192.168.0.222}{192.168.0.222:9300}, {escluster-es1}{WELN6zJ8QYy40zew1Axs2w}{Zc4W94tbT7exlRHziAnd-g}{192.168.0.221}{192.168.0.221:9300}]: NoNodeAvailableException[None of the configured nodes were available: [{escluster-es2}{-iqECyceSn2FBSPbg-nFvw}{F2PCjdZTTUujzj5996um6w}{192.168.0.222}{192.168.0.222:9300}, {escluster-es1}{WELN6zJ8QYy40zew1Axs2w}{Zc4W94tbT7exlRHziAnd-g}{192.168.0.221}{192.168.0.221:9300}]]; nested: NodeDisconnectedException[[escluster-es1][192.168.0.221:9300][indices:data/read/search] disconnected];
at org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:266)
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51)
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:955)
at org.elasticsearch.transport.TransportService$Adapter.lambda$raiseNodeDisconnected$3(TransportService.java:783)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:451)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: NodeDisconnectedException[[escluster-es1][192.168.0.221:9300][indices:data/read/search] disconnected]

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.