Hey Everyone,
I have an elasticsearch cluster set up on amazon's elastic beanstalk
infrastructure; it's deployed in a tomcat7 container, and using
elasticsearch.wares to fire up a node. my configuration is pretty much from
this tutorial.
I have been load testing the system, and periodically, I get a
"NoNodeAvailable" exception. I turned on debug level logging for
org.elasticsearch, and here's what I see. I've implemented some retry logic
as a band-aid, but it doesn't fix the problem 100%, and it feels like a
band aid.
I haven't been able to reproduce this problem locally. Does anyone have an
idea of what might be happening here?
14:47:37,155 INFO fetch:104 - [Angel Dust] [content][4] took[61.4micros],
took_millis[0], search_type[QUERY_THE
N_FETCH], total_shards[5],
source[{"from":0,"size":100,"query":{"bool":{"must":{"bool":{"should":{"match":{"body
":{"query":"biscuits","type":"boolean"}}}}}}}}], extra_source[],
14:47:37,171 DEBUG netty:94 - [Speed Demon] disconnected from
[[#transport#-1][inet[redowl-refdata-service.elast
icbeanstalk.com/23.23.118.78:9300]]]
14:47:37,295 INFO SearchContentResource:114 - Exception happened whilst
searching for
actors: [],
keywords [enron],
eventStartInSeconds: null,
durationInSeconds: null
java.util.concurrent.ExecutionException:
org.elasticsearch.client.transport.NoNodeAvailableException: No node av
ailable
at
org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:285)
at
org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:272)
I then see some successful queries on different nodes:
4:47:37,387 INFO query:104 - [Angel Dust] [content][3] took[8.8ms],
took_millis[8], search_type[QUERY_THEN_FETCH], total_shards[5],
source[{"from":0,"size":100,"query":{"bool":{"must":{"bool":{"should":{"match":{"body":{"query":"enron","type":"boolean"}}}}}}}}],
extra_source[],
and then, a reconnect:
*14:47:38,200 DEBUG netty:94 - [Speed Demon] connected to node
[[#transport#-1][inet[redowl-refdata-service.elasticbeanstalk.com/23.23.118.78:9300]]]
*
Thanks for any thoughts you may have. Elastic Search is working fantastic
for most cases, it's just this one error we uncovered during load testing.
Best,
--paul
--