We're experiencing a problem where the client constantly throws an
IndexMissingException when searching after rejoining the cluster after a
network failure. The server log shows that the client is removed after
pinging it failed. When the network comes up again the log shows that
the client is added as expected but the client starts to throw
IndexMissingException when searching in an index that is known to exist.
I've managed to create a simple test that reproduces this problem. Here
are the steps:
Start an ES 0.19.2 server instance (default config):
Start the attached Java program on the same machine as the server. I'm
using the attached elasticsearch.yml. The code will create the index
'estest' and then run a match_all search every 2 seconds.
Drop all network traffic on ports 9300 and 9301 to simulate a network
sudo iptables -A INPUT -p tcp --destination-port 9300 -j DROP ; sudo
iptables -A INPUT -p tcp --destination-port 9301 -j DROP
Wait for the server and client to give up on pinging the other side.
Start network traffic again:
sudo iptables --flush
Client and server will appear to find each other again but then the
client will start to throw IndexMissingException for every search.
It would be great if someone else could try this too and let me know if
it can be reproduced or if it's only me. I'm not not sure using iptables
like I'm doing is a good way to simulate the network problems though
we're seeing the exact same symptoms when there's a real network problem
in our production environment. In production the client and server are
on different machines.