How to restore elasticsearch s3 snapshot on a different cluster

[Kay_Khan]

I have an Elastic Cloud on Kubernetes (ECK) setup and i have also setup snapshots which are stored on s3.

I want to restore these snapshots on s3 to a local version of ECK i have running on minikube. So how do i restore snapshots that are on s3 to a different (local) elastic search.

[warkolm (Mark Walkom)]

You need to add the snapshot repo to the other cluster, then you can restore from it - https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-register-repository.html

[Kay_Khan]

Thanks that was quite simple.

A follow up question, i have setup a local cluster on my machine usng ECK and restored a single index. However the cluster has gone into red and i noticed my index has an empty docs count value, what happend here

I have 3 pods

1 of the pods shows the followings logs

low disk watermark [85%] exceeded on [xGU-cJhOSGGm-R2svSdqYA][data-es-es-data-1][/usr/share/elasticsearch/data/nodes/0] free: 14.3gb[14.6%], replicas will not be assigned to this node

{"type": "server", "timestamp": "2020-07-15T09:21:43,966Z", "level": "INFO", "component": "o.e.c.r.a.DiskThresholdMonitor", "cluster.name": "data-es", "node.name": "data-es-es-data-2", "message": "low disk watermark [85%] exceeded on [4Ef_g1I_R0yQg_y9TSOSJA][data-es-es-data-0][/usr/share/elasticsearch/data/nodes/0] free: 14.3gb[14.6%], replicas will not be assigned to this node", "cluster.uuid": "15z2BC1JSWCGIyvmDkHMXg", "node.id": "d42rGeFvTsGnCDhnvQrK-A"  }
{"type": "server", "timestamp": "2020-07-15T09:21:46,349Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "data-es", "node.name": "data-es-es-data-2", "message": "Cluster health status changed from [YELLOW] to [RED] (reason: [async_shard_fetch]).", "cluster.uuid": "15z2BC1JSWCGIyvmDkHMXg", "node.id": "d42rGeFvTsGnCDhnvQrK-A"  }
{"type": "server", "timestamp": "2020-07-15T09:22:13,979Z", "level": "INFO", "component": "o.e.c.r.a.DiskThresholdMonitor", "cluster.name": "data-es", "node.name": "data-es-es-data-2", "message": "low disk watermark [85%] exceeded on [xGU-cJhOSGGm-R2svSdqYA][data-es-es-data-1][/usr/share/elasticsearch/data/nodes/0] free: 14.3gb[14.6%], replicas will not be assigned to this node", "cluster.uuid": "15z2BC1JSWCGIyvmDkHMXg", "node.id": "d42rGeFvTsGnCDhnvQrK-A"  }
{"type": "server", "timestamp": "2020-07-15T09:22:13,979Z", "level": "INFO", "component": "o.e.c.r.a.DiskThresholdMonitor", "cluster.name": "data-es", "node.name": "data-es-es-data-2", "message": "low disk watermark [85%] exceeded on [d42rGeFvTsGnCDhnvQrK-A][data-es-es-data-2][/usr/share/elasticsearch/data/nodes/0] free: 14.3gb[14.6%], replicas will not be assigned to this node", "cluster.uuid": "15z2BC1JSWCGIyvmDkHMXg", "node.id": "d42rGeFvTsGnCDhnvQrK-A"  }

And the other 2 have the following

waiting for elected master node

{"type": "server", "timestamp": "2020-07-15T09:21:46,459Z", "level": "INFO", "component": "o.e.x.m.e.l.LocalExporter", "cluster.name": "data-es", "node.name": "data-es-es-data-0", "message": "waiting for elected master node [{data-es-es-data-2}{d42rGeFvTsGnCDhnvQrK-A}{-g6ho5RoQl-IlJQUHTOcNQ}{172.18.0.9}{172.18.0.9:9300}{dilm}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)", "cluster.uuid": "15z2BC1JSWCGIyvmDkHMXg", "node.id": "4Ef_g1I_R0yQg_y9TSOSJA"  }
{"type": "server", "timestamp": "2020-07-15T09:21:56,093Z", "level": "INFO", "component": "o.e.x.m.e.l.LocalExporter", "cluster.name": "data-es", "node.name": "data-es-es-data-0", "message": "waiting for elected master node [{data-es-es-data-2}{d42rGeFvTsGnCDhnvQrK-A}{-g6ho5RoQl-IlJQUHTOcNQ}{172.18.0.9}{172.18.0.9:9300}{dilm}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)", "cluster.uuid": "15z2BC1JSWCGIyvmDkHMXg", "node.id": "4Ef_g1I_R0yQg_y9TSOSJA"  }
{"type": "server", "timestamp": "2020-07-15T09:21:57,138Z", "level": "INFO", "component": "o.e.x.m.e.l.LocalExporter", "cluster.name": "data-es", "node.name": "data-es-es-data-0", "message": "waiting for elected master node [{data-es-es-data-2}{d42rGeFvTsGnCDhnvQrK-A}{-g6ho5RoQl-IlJQUHTOcNQ}{172.18.0.9}{172.18.0.9:9300}{dilm}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)", "cluster.uuid": "15z2BC1JSWCGIyvmDkHMXg", "node.id": "4Ef_g1I_R0yQg_y9TSOSJA"  }

[Kay_Khan]

I fixed the watermark error with the following https://stackoverflow.com/questions/33369955/low-disk-watermark-exceeded-on

curl -XPUT localhost:9200/_cluster/settings -d '{
    "transient" : {
        "cluster.routing.allocation.disk.threshold_enabled" : false
    }
}'

Still unsure about the elected masters issue...