Hi,
We had an incident where all master nodes failed. After they were auto-replaced the old data nodes did not reconnect to the master due to state mismatch. While we were able to clear down the state so the nodes could rejoin the cluster we were not able to recover the index data.
How can we recover index data from data nodes that are effectively joining a new cluster?
There's no safe way to do that. The master nodes hold the cluster metadata, without which the data on the data nodes is kinda meaningless. The best way forwards is to restart the failed master nodes, but if you can't do that you should recover your data from a recent snapshot.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.