Master nodes lost

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hello Users + Experts.

Due to unfortunate operator error the PVs of the master nodes were deleted, due to a out of disk error from the master nodes.

We had 3 dedicated master nodes and 12 data nodes.
The data nodes are now can not locate the master after provisioning new master nodes (and yes that node set was scaled down to zero sadly).

{"type": "server", "timestamp": "2020-07-29T14:25:20,697Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "eck-prod", "node.name": "eck-prod-es-data-nodes-2-3000-0", "message": "path: /_ilm/policy/filebeat-7.4.1, params: {name=filebeat-7.4.1}", "cluster.uuid": "vcExmqmWQqKJjBuIQ8b1TA", "node.id": "9CIgWcavQXedBXGEHPfpqA" , 
"stacktrace": ["org.elasticsearch.discovery.MasterNotDiscoveredException: null",
"at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:218) [elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.5.2.jar:7.5.2]",
"a

We are therefore left with 12 data nodes (from 4 node sets) and no ability to form a cluster.
We have tried a few things to recover the cluster
Following this guide: https://www.elastic.co/guide/en/elasticsearch/reference/current/node-tool.html
Which is clearly not applicable to kubernetes and eck.

We have additionally thought about making 3 of the data nodes master eligible, and if possible triggering a master election... although we aren't sure if this is the right path.

Any help or guidance appreciated!!

2

By far the best option would be to restore from a snapshot into a new cluster. If that is not possible you could try the following:

  • spin up new master nodes (I think you have done that already)
  • set cluster.initial_master_nodes explicitly to the new names of the new master nodes (this will generate a warning in the logs as this setting is usually managed by ECK)
  • now for the dangerous part. Please note that the following can lead to data loss and should only be your measure of last resort:
    • convince yourself that you have correctly configured cluster.initial_master_nodes and the new masters have formed a new cluster
    • now edit the spec of your data nodes adding an init container that will detach the data nodes from the old cluster and allow them to join the newly formed cluster. Again this is where you can lose data. I used the following container in an experiment:
      spec:
        initContainers:
        - name: detach
          command:
          - sh
          - -c
          - |
            printf "y\n" | bin/elasticsearch-node detach-cluster
  • Make sure that the new pod specification with the addtional init container is used in the corresponding stateful set. If not wait for the operator to update the stateful set before you continue.
  • Now start with one of the data nodes and delete the corresponding pod. The pod will then be recreated by the stateful set controller using the new spec.
  • Check the pods logs and verify that it successfully joins the new cluster before proceding to repeat the same procedure with the other data pods.
  • Once all nodes have joined the new cluster remove the init container (important otherwise the pods will detach every time they restart) and also the cluster.initial_master_nodes setting again.

I suggest to go through the procedure on a test cluster first to verify it works for your setup, before you try to apply to the cluster that has lost its master nodes.

[EDIT] I have simplified the instructions a bit. The cluster.initial_master_nodes setting is not necessary on the data nodes.

3

Many thanks Peter,
We don't snapshot globally or any system indices, only specific indices, so that option is out.

I had thought about the initial master node setting as well, but the detach cluster on initContainers was the missing ingredient - genius!

-marcus