Manual backup restore

Elastic Stack Elasticsearch
1

Greetings,

We’re currently facing an issue with our Elasticsearch cluster, which consists of 8 nodes. Unfortunately, we accidentally “emptied” an index (my_index) from the cluster. While we don’t have a snapshot, we do have a backup of the entire Elasticsearch data folder. We attempted to restore the index data manually by copying it back into the appropriate UUID directory in the data folder.

However, after doing this and restarting the Elasticsearch service, the index is being automatically rewritten by the cluster with new data.

What We’ve Tried:

  • We excluded one of the nodes, "node-8_", from the cluster and tried pointing it to the data folder. Elasticsearch started, but we were unable to log in or reset the password. The error message kept saying the cluster health is not determined.
  • We also tried setting the discovery.seed to single-node mode, but this also didn’t work.

Current Situation:

The goal is to restore the original index data (my_index) and merge it with the new data that is currently being written to the index. Essentially, we want to retain the existing data from the backup and integrate it with the fresh data that is already being written to my_index.

My Question

Is there any way around this issue to restore the my_index and merge the old data with the new data?

Any help or advice on how to approach this would be greatly appreciated!

Thank you!

2

Hi @iexpertini , let’s give this a shot—I’m not sure it’ll work, but here’s what I’d try:

  1. Spin up a single-node ES instance using the same version (e.g. 8.x).

  2. Copy your entire backup data/ directory into that node’s path.data.

  3. In elasticsearch.yml, set:

cluster.name: restore-cluster
node.name: restore-node
path.data: /path/to/backup/data
discovery.type: single-node
xpack.security.enabled: false
network.host: 0.0.0.0
http.port: 9200

  1. Start Elasticsearch—this node should pick up the old cluster state and bring back my_index.

  2. Verify by hitting:

GET http://restore-node:9200/_cat/indices?v
GET http://restore-node:9200/my_index/_count

I hope you can fix your problem soon.

3

Do you know the number of primary / replica shards for that index, and where they were across your 8 nodes before you deleted the documents from the index?

IMO your best bet is to create a new 8-node cluster, maybe using VMs, copy your backup data into that new cluster, try to get your data back there.

If that succeeds, you can worry about how to merge with the "real" index in the production cluster.