Elasticsearch Nonodefoundexception issue


(sunny) #1

Our spring boot app is indexing data in ES cluster using transport client [using ElasticsearchRepository ]. Issue is that after some index call indexing just hangs the.the SAVE method on DAO just hangs and After some time it gives us "Nonodefoundexception"

Here’s our stack - ES 1.7.1 - Spring Boot: 1.3.2.RELEASE - Java: 1.8 - spring-boot-starter-data-elasticsearch : 1.3.2.RELEASE

Using transport client:

ES setup: - there are three nodes. all are client and master and data

Environment: Our dev is in AWS and we do not see any issue in Dev even though all ES setup is same.

org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes were available: [[NodeName-apsrp5470][TzlZ3u8aTK23YFr14R085g][apsrp5470][inet[/10.222.146.127:9300]]{master=true}, [NodeName-apsrp5434][K4r0wujuThqwj1oTU9lEHg][apsrp5434][inet[/10.222.146.132:9300]]{master=true}, [NodeName-apsrp5435][YBPk8E4TR6u7LFLxfaVWaw][apsrp5435][inet[/10.222.146.139:9300]]{master=true}] at org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:242) ~[elasticsearch-1.5.2.jar!/:na] at org.elasticsearch.action.TransportActionNodeProxy$1.handleException(TransportActionNodeProxy.java:78) ~[elasticsearch-1.5.2.jar!/:na] at org.elasticsearch.transport.TransportService$Adapter$3.run(TransportService.java:468) ~[elasticsearch-1.5.2.jar!/:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_60] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] Caused by: org.elasticsearch.transport.NodeDisconnectedException: [NodeName-apsrp5470][inet[/10.222.146.127:9300]][indices:data/write/index] disconnected

Thread Dumps :- java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007bd824f48> (a org.elasticsearch.common.util.concurrent.BaseFuture$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:279) at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:118) at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:45) at org.springframework.data.elasticsearch.core.ElasticsearchTemplate.index(ElasticsearchTemplate.java:519)


(system) #2