NoNodeAvailableException: No node available on Elastic Beanstalk

Elastic Stack Elasticsearch
1

Hello,
I have a cluster set up in AWS, and it looks fine.
Now, my application, which connects to ES via TransportClient is
running on ElasticBeanstalk.
I have set client.transport.sniff=true and cluster name is set as
well.
I don't set any inetSockerTransportAddresses
Once my app is started I'm getting
org.elasticsearch.client.transport.NoNodeAvailableException: No node
available
at
org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:
170)
at
org.elasticsearch.client.transport.support.InternalTransportClient.search(InternalTransportClient.java:
254)

If I set sniff to false and provide local IP addresses of the boxes
I'm running ES on, I'm getting the same exception.
But, if I set it to public URL as inetSockerTransportAddresses it
works.

Honestly, I don't understand hot sniff should find my cluster in the
first place.
Please, any information is appreciated.

Thank you,
E.S.

2

sniff will not go and find the cluster automatically. You need to add the
address of at least one node in the cluster to the transport client. What
sniff will do when enabled, is that it will get from that node the rest of
the nodes in the cluster, and use all of those when round robin requests.

On Thu, Dec 1, 2011 at 2:38 AM, Eugene Strokin eugene@strokin.info wrote:

Hello,
I have a cluster set up in AWS, and it looks fine.
Now, my application, which connects to ES via TransportClient is
running on ElasticBeanstalk.
I have set client.transport.sniff=true and cluster name is set as
well.
I don't set any inetSockerTransportAddresses
Once my app is started I'm getting
org.elasticsearch.client.transport.NoNodeAvailableException: No node
available
at

org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:
170)
at

org.elasticsearch.client.transport.support.InternalTransportClient.search(InternalTransportClient.java:
254)

If I set sniff to false and provide local IP addresses of the boxes
I'm running ES on, I'm getting the same exception.
But, if I set it to public URL as inetSockerTransportAddresses it
works.

Honestly, I don't understand hot sniff should find my cluster in the
first place.
Please, any information is appreciated.

Thank you,
E.S.

3

I c... Thank you. I guess it would make sense to get a static IP for
one of the nodes of the cluster, and set it in properties of the
application. If the ES server would get restarted, I wouldn't need to
change the IP, and the rest nodes will be sniffed even if IPs will be
changed.

On Dec 1, 9:10 am, Shay Banon kim...@gmail.com wrote:

sniff will not go and find the cluster automatically. You need to add the
address of at least one node in the cluster to the transport client. What
sniff will do when enabled, is that it will get from that node the rest of
the nodes in the cluster, and use all of those when round robin requests.

On Thu, Dec 1, 2011 at 2:38 AM, Eugene Strokin eug...@strokin.info wrote:

Hello,
I have a cluster set up in AWS, and it looks fine.
Now, my application, which connects to ES via TransportClient is
running on ElasticBeanstalk.
I have set client.transport.sniff=true and cluster name is set as
well.
I don't set any inetSockerTransportAddresses
Once my app is started I'm getting
org.elasticsearch.client.transport.NoNodeAvailableException: No node
available
at

org.elasticsearch.client.transport.TransportClientNodesService.execute(Tran sportClientNodesService.java:
170)
at

org.elasticsearch.client.transport.support.InternalTransportClient.search(I nternalTransportClient.java:
254)

If I set sniff to false and provide local IP addresses of the boxes
I'm running ES on, I'm getting the same exception.
But, if I set it to public URL as inetSockerTransportAddresses it
works.

Honestly, I don't understand hot sniff should find my cluster in the
first place.
Please, any information is appreciated.

Thank you,
E.S.

4

I have a similar problem. Using AWS and ElasticBeanstalk on one small
node (for the moment). The node is up & running and I'm using a simple
Tomcat WAR to interface to the cluster. I get a
NoNodeAvailableException using this:

client = new TransportClient(settingsBuilder()
.put("cluster.name", "example-cluster")
.put("client.transport.sniff", true)
.build())
.addTransportAddress(new
InetSocketTransportAddress("localhost", 9300));

If I ssh into the EC2 instance, a curl'd health check to localhost:
9200 works fine and a netstat -an gives me

tcp 0 0 :::
9300 :::* LISTEN

so it seems like this should work. Suggestions?

Thanks

On Dec 1, 7:10 am, Shay Banon kim...@gmail.com wrote:

sniff will not go and find the cluster automatically. You need to add the
address of at least one node in the cluster to the transport client. What
sniff will do when enabled, is that it will get from that node the rest of
the nodes in the cluster, and use all of those when round robin requests.

5

The short answer is "never mind".

The longer answer is that the exception was being thrown while
preparing a search with an invalid term (using termQuery), which threw
me off the scent.

On Dec 4, 4:45 pm, Gramby sp...@splix.org wrote:

I have a similar problem. Using AWS and ElasticBeanstalk on one small
node (for the moment). The node is up & running and I'm using a simple
Tomcat WAR to interface to the cluster. I get a
NoNodeAvailableException using this:

client = new TransportClient(settingsBuilder()
.put("cluster.name", "example-cluster")
.put("client.transport.sniff", true)
.build())
.addTransportAddress(new
InetSocketTransportAddress("localhost", 9300));

If I ssh into the EC2 instance, a curl'd health check to localhost:
9200 works fine and a netstat -an gives me

tcp 0 0 :::
9300 :::* LISTEN

so it seems like this should work. Suggestions?

Thanks

On Dec 1, 7:10 am, Shay Banon kim...@gmail.com wrote:

6

In the interest of not misleading anyone based on my previous
message...

The real problem that I had was that I was doing a client.close() at
the end of my jsp file but only creating the connection if(client ==
null). Since I wasn't setting client to null after the close, the
first time the jsp was rendered it worked; subsequent times didn't.
And since this was on Elastic Beanstalk, the health monitor was eating
my one working session. I just happened to get lucky (unlucky?) and
get the first execution when I was messing with the terms in my query.

My solution was to make my client object a static (I suppose I should
make it final, too) and never close it since the transport client acts
similarly to a connection pool.

Sorry if my previous caused any confusion.

On Dec 4, 7:26 pm, Gramby sp...@splix.org wrote:

The short answer is "never mind".

The longer answer is that the exception was being thrown while
preparing a search with an invalid term (using termQuery), which threw
me off the scent.

On Dec 4, 4:45 pm, Gramby sp...@splix.org wrote:

7

Gramby, Since you have experience with Beanstack, here is another
problem which you could know how to solve:
I have the same, typical situation: application which uses ES via
TransferClient on Beanstalk, and ES Cluster on other AWS instances. I
have a singleton which provides the client to the application, so this
is something similar like you have it static. I nether close the
client.
Once I deploy a new version to Beanstalk, I'm getting NodeNotFound
exception. This is strange, because nothing is changed on ES side at
all.
I don't know how to solve it really, I'm just trying different things.
So I found out the only way it starts working, when I do following
steps:

  • Deploy new version of the application on Beanstalk, start getting
    NodeNotFound exception
  • Restart ES
  • Open ports 9300 and 9200 to public on ES security policy (I don't
    know if it affects the connection inside of the AWS network or not),
    I'm doing it only to check ES from my local computer, so I'm not sure
    if it affects anything.
  • Wait 5 minutes while ES pics up the index files from S3
  • Everything starts working
  • Close ports 9300 and 9200 to public, everything works as expected.

So, I guess my questions would be: Do you have NodeNotFound problem
when you deploy a new version of the application? Or could you suggest
something about working with Beanstalk.
Apparently, Beanstalk has downtime during deployment even without any
problems, and this problem with the connection makes it even worse.

Thank you,
any comments or suggestions are appreciated,
Eugene S.

On Dec 4, 10:55 pm, Gramby sp...@splix.org wrote:

In the interest of not misleading anyone based on my previous
message...

The real problem that I had was that I was doing a client.close() at
the end of my jsp file but only creating the connection if(client ==
null). Since I wasn't setting client to null after the close, the
first time the jsp was rendered it worked; subsequent times didn't.
And since this was on Elastic Beanstalk, the health monitor was eating
my one working session. I just happened to get lucky (unlucky?) and
get the first execution when I was messing with the terms in my query.

My solution was to make my client object a static (I suppose I should
make it final, too) and never close it since the transport client acts
similarly to a connection pool.

Sorry if my previous caused any confusion.

On Dec 4, 7:26 pm, Gramby sp...@splix.org wrote:

The short answer is "never mind".

The longer answer is that the exception was being thrown while
preparing a search with an invalid term (using termQuery), which threw
me off the scent.

On Dec 4, 4:45 pm, Gramby sp...@splix.org wrote:

8

Hi Eugene,

I'm not sure if I can pinpoint exactly what your problem is, but based
on your original message and this one, I suspect you have a firewall
problem. Let me make sure I understand this correctly:

  1. You have ES running on an EC2 cluster of one or more instances.
  2. You're using an Elastic Beanstalk instance to connect to that
    cluster using a fixed IP or name of some sort for the cluster.
  3. When you deploy a new WAR to your Elastic Beanstalk instance, you
    can't seem to connect to ES on your EC2 cluster anymore.

Is this correct?

I'm assuming you've set up a security group for your cluster so that
all of your nodes can talk to each other. Within that group, have you
added your Elastic Beanstalk security group as a source for inbound
traffic to your cluster on ports 9200 and 9300? You don't have to just
specify a specific IP for the inbound traffic; you can also use other
security groups as a source.

Also, if you SSH into one of your cluster nodes, you should be able to
verify that ES is still running using:

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

(this assumes that you're starting your nodes with IP address 0.0.0.0)

I also added the public IP address of my router to my security group
for ports 9200 & 9300 so that I can access the cluster without
shelling into it or making the ports open to the world.

Hope this helps.

Also, feel free to email me directly if you need a quicker response.

On Dec 5, 8:32 pm, Eugene Strokin eug...@strokin.info wrote:

Gramby, Since you have experience with Beanstack, here is another
problem which you could know how to solve:
I have the same, typical situation: application which uses ES via
TransferClient on Beanstalk, and ES Cluster on other AWS instances. I
have a singleton which provides the client to the application, so this
is something similar like you have it static. I nether close the
client.
Once I deploy a new version to Beanstalk, I'm getting NodeNotFound
exception. This is strange, because nothing is changed on ES side at
all.
I don't know how to solve it really, I'm just trying different things.
So I found out the only way it starts working, when I do following
steps:

  • Deploy new version of the application on Beanstalk, start getting
    NodeNotFound exception
  • Restart ES
  • Open ports 9300 and 9200 to public on ES security policy (I don't
    know if it affects the connection inside of the AWS network or not),
    I'm doing it only to check ES from my local computer, so I'm not sure
    if it affects anything.
  • Wait 5 minutes while ES pics up the index files from S3
  • Everything starts working
  • Close ports 9300 and 9200 to public, everything works as expected.

So, I guess my questions would be: Do you have NodeNotFound problem
when you deploy a new version of the application? Or could you suggest
something about working with Beanstalk.
Apparently, Beanstalk has downtime during deployment even without any
problems, and this problem with the connection makes it even worse.

Thank you,
any comments or suggestions are appreciated,
Eugene S.

On Dec 4, 10:55 pm, Gramby sp...@splix.org wrote: