We have faced a problem in our live environment, which caused index mapping process to fail. Let me tell you exactly when the exception was coming.
We are using 2 node cluster (Elastic 1.4.1, 4 core, 8gb each) in our LIVE env.
Our application creates one index for one virtual event (I know that's too much, but, you know, can't touch legacy code). One day, the event creation process stopped working. After some investigation, we found that something was wrong with Index Creation process. After looking further, we found that index were getting created, but there were some problems with executing _mapping API. According to our logs, response of _mapping API was ProcessClusterEventTimeoutException.
We faced this problem for quite some time, until we chose to remove one node and restart the server. I still don't know, what might have caused it.
I would really love to avoid these kind of problems in future. Can you tell me what are the possible causes of ProcessClusterEventTimeoutException, and how can I debug one if I encounter it next time?