TransportClient and NoNodeAvailableException

Elastic Stack Elasticsearch
1

I have an application (a java jersey based REST service) that uses an
ElasticSearch index; it fetches data out of ElasticSearch using a
TransportClient to use as part of the data its serving.

My problem is that I have periodically gotten NoNodeAvailable exceptions
when doing searches: there is retry logic and the query usually succeeds,
but the retry costs me something in latency, and also just bothers me that
I don't understand what's going on with the exceptions. I think the root
cause may be in the way that I am using a transport client.

The way the application is currently built, we're using spring to
instantiate our DAO classes, and each of these has a reference to a
transport client. The transport client is re-used througout the life of the
application. Should I instead be instantiating a new TransportClient on
each search request? Does this matter?

If it matters, I've reproduced this problem on a 1 node cluster, where the
client and the server are both on running on the same box. I use a
relatively small amount of data in the index (1GB).

any thoughts or help would be much appreciated! Here's the stacktrace I get:

org.elasticsearch.client.transport.NoNodeAvailableException: No node
available
at
org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:246)
at
org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:214)
at
org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:106)
at
org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:210)
at
org.elasticsearch.client.transport.TransportClient.search(TransportClient.java:388)
at
org.elasticsearch.action.search.SearchRequestBuilder.doExecute(SearchRequestBuilder.java:816)
at
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:62)
at
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:57)

at
com.redowlanalytics.reference.dao.SearchIndexDAO.getSearchResponse(SearchIndexDAO.java:485)

--paul

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Do you use sniffing? And how many interface addresses do you connect to?
Is the cluster name correct?
I experienced this when TransportClient is set to snffing but ES cluster
could not be reachable at all localhost/IPv4/IPv6 networks. One time I
got disconnects because of cluster name mismatch or when cluster did not
respond within 5 seconds, but that is easy to find out. Maybe you can
try to turn sniffing off and focus on a single NIC only to be sure where
the disconnect happens.

Jörg

Am 08.05.13 19:23, schrieb Paul Sanwald:

org.elasticsearch.client.transport.NoNodeAvailableException: No node
available
at
org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:246)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Hi Jörg,
I've pared it down to a single node, the cluster name is definitely
correct as this works most of the time; I only get this error occasionally.
I do have sniffing turned on, but could try disabling it.

for what it's worth, I've never been able to reproduce this on my local
machine, so I suspect it might have something to do with discovery or the
gateway, which I have configured as follows:
discovery.type: ec2
gateway.type: s3
gateway.s3.bucket: ${elastic.search.s3.bucket}
discovery.ec2.groups: ${elastic.search.discovery.group}

--paul

On Wednesday, May 8, 2013 5:22:17 PM UTC-4, Jörg Prante wrote:

Do you use sniffing? And how many interface addresses do you connect to?
Is the cluster name correct?
I experienced this when TransportClient is set to snffing but ES cluster
could not be reachable at all localhost/IPv4/IPv6 networks. One time I
got disconnects because of cluster name mismatch or when cluster did not
respond within 5 seconds, but that is easy to find out. Maybe you can
try to turn sniffing off and focus on a single NIC only to be sure where
the disconnect happens.

Jörg

Am 08.05.13 19:23, schrieb Paul Sanwald:

org.elasticsearch.client.transport.NoNodeAvailableException: No node
available
at

org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:246)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

I'm not using Amazon EC2 but I remember it might help to increase the
default "ping.timeout" parameter from 5s up to a higher value.

Jörg

Am 09.05.13 21:43, schrieb Paul Sanwald:

discovery.type: ec2
gateway.type: s3
gateway.s3.bucket: ${elastic.search.s3.bucket}
discovery.ec2.groups: ${elastic.search.discovery.group}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

do you mean
discovery.zen.ping_timeout, or client.transport.ping_timeout?

I would assume the latter, but wanted to verify.

--paul

On Thursday, May 9, 2013 3:53:31 PM UTC-4, Jörg Prante wrote:

I'm not using Amazon EC2 but I remember it might help to increase the
default "ping.timeout" parameter from 5s up to a higher value.

Jörg

Am 09.05.13 21:43, schrieb Paul Sanwald:

discovery.type: ec2
gateway.type: s3
gateway.s3.bucket: ${elastic.search.s3.bucket}
discovery.ec2.groups: ${elastic.search.discovery.group}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

client.transport.ping_timeout

If that does not help, you should check what happened to the node you
try to connect to (OOM, disk full, long GC, just to name a few
catastrophic events)

Jörg

Am 09.05.13 22:07, schrieb Paul Sanwald:

do you mean
discovery.zen.ping_timeout, or client.transport.ping_timeout?

I would assume the latter, but wanted to verify.

--paul

On Thursday, May 9, 2013 3:53:31 PM UTC-4, Jörg Prante wrote:

I'm not using Amazon EC2 but I remember it might help to increase the
default "ping.timeout" parameter from 5s up to a higher value.

Jörg

Am 09.05.13 21:43, schrieb Paul Sanwald:
> discovery.type: ec2
> gateway.type: s3
> gateway.s3.bucket: ${elastic.search.s3.bucket}
> discovery.ec2.groups: ${elastic.search.discovery.group}

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.