Elasticsearch/X-Pack 5.5.2
I created a prototype cluster with 4 data nodes and 1 master. X-pack was installed on every node and everything was working correctly. The cluster had been running for several weeks and about a week with X-Pack. A decision was made to convert the existing master to a coordinating node (node also had kibana, cerebros, and mirage installed on it). Thus I decided to create a new master node and convert the existing master to a coordinating node.
The new master node was installed with X-Pack and configured as the master. At this point all nodes in the cluster were stopped and nodes were started one at a time, starting with the new master and then the coordinating node (old master), which worked correctly. However, none of the data nodes would join the cluster. The log entries in the master indicated that a ping response was not received, yet I could see the configuration version change entries on the data node. I then stopped all nodes and converted the coordinating node back to the master. All data nodes were still unable to connect due to the same issue with the master not receiving a ping response. After changing the logging level to debug on all of the nodes, I suspected that X-Pack was causing a problem. I first removed X-Pack from the original master but the data node still did not respond from a ping request. Upon removing X-Pack from a data node it was able to respond to a ping request and join the cluster. Thus I had to remove X-Pack from every node to get the cluster back up.
After restarting all nodes, I discovered that all of the templates were deleted. X-Pack created several templates and I can understand why these were deleted, however, there were several templates created for our deployment that were also deleted. None of the indices or aliases for the deployment were affected.
Basically, I had two problems:
- data nodes unable to join cluster after creating a new master
- deletion of templates after removing X-Pack
Not sure if this is an X-Pack issue or an Elasticsearch issue.