Below are my observations with ES cluster 3 nodes/instances with ES 2.1 version .
we have some client application (say rest services) establishes ES client connections with ES cluster 3 nodes during server startup . I see all connections ESTABLISHED with those 3 ES nodes IP addresses with command netstat -an | grep 9300
when I rebuild ES node one and waited until new ES node is built with new IP and is added back to cluster with green health then followed to rebuild other 2 nodes similarly . Rebuilding ES nodes one after one - now 3 nodes having 3 new IP addresses
client.transport.sniff = false
I see NoRouteToHostException being logged continuously. when the last node is rebuilt I see error "org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{
" and also netstat -an | grep 9300 command NOT showing connections with new ES IP addresses. It still shows old IP connections.I dont see my Rest service application able to perform CRUD operations on ES new nodes.
To fix this I need to restart my application to get new connections with new ES nodes.
client.transport.sniff=true
NoRouteToHostException is being logged continuously. But I did not see org.elasticsearch.client.transport.NoNodeAvailableException.
I am able to perform CRUD operations from my client application and also netstat -an | grep 9300 command shows connections with new ES IP addresses
NoRouteToHostException is in this case (client.transport.sniff=true) is just WARN message ? can we ignore ? please suggest