NoNodeAvailable exception: amazon EBS deployment of elasticsearch

(Paul Sanwald) #1

Hey Everyone,
I have an elasticsearch cluster set up on amazon's elastic beanstalk
infrastructure; it's deployed in a tomcat7 container, and using
elasticsearch.wares to fire up a node. my configuration is pretty much from
this tutorial.

I have been load testing the system, and periodically, I get a
"NoNodeAvailable" exception. I turned on debug level logging for
org.elasticsearch, and here's what I see. I've implemented some retry logic
as a band-aid, but it doesn't fix the problem 100%, and it feels like a
band aid.

I haven't been able to reproduce this problem locally. Does anyone have an
idea of what might be happening here?

14:47:37,155 INFO fetch:104 - [Angel Dust] [content][4] took[61.4micros],
took_millis[0], search_type[QUERY_THE
N_FETCH], total_shards[5],
":{"query":"biscuits","type":"boolean"}}}}}}}}], extra_source[],
14:47:37,171 DEBUG netty:94 - [Speed Demon] disconnected from
14:47:37,295 INFO SearchContentResource:114 - Exception happened whilst
searching for
actors: [],
keywords [enron],
eventStartInSeconds: null,
durationInSeconds: null
org.elasticsearch.client.transport.NoNodeAvailableException: No node av

I then see some successful queries on different nodes:
4:47:37,387 INFO query:104 - [Angel Dust] [content][3] took[8.8ms],
took_millis[8], search_type[QUERY_THEN_FETCH], total_shards[5],

and then, a reconnect:

*14:47:38,200 DEBUG netty:94 - [Speed Demon] connected to node

Thanks for any thoughts you may have. Elastic Search is working fantastic
for most cases, it's just this one error we uncovered during load testing.




(system) #2