Hello,
I've tested backup and restore of ELK stack deployed with Elastic operator and I have problem with restoring Elasticsearch.
I've opened an issue on GitHub as it seems to me it's an ECK bug:
In any case, I could also misconfigure something.
Below are the details of the problem:
What did you do?
I've created a schedule with a daily Kubernetes backup in Velero (without Volumes' data). Then I've removed elastic operator, ELK and filebeat for test:
$ kubectl delete ns elastic-system elk-stack filebeat
And then I restored it from the backup:
$ velero restore create --from-backup kubernetes-daily-20200330000002 --include-namespaces elastic-system,elk-stack,filebeat
The restore is successful, but elasticsearch afterwards is not working.
Here're the logs:
https://pastebin.com/R8DpUrpH
It seems to me that this is core of the problem:
{"type": "server", "timestamp": "2020-03-31T12:52:40,008Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elk-stack", "node.name": "elk-stack-es-master-green-0", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{elk-stack-es-master-green-0}{TWEH9Px6TBm92a9dL-mRgg}{pNPr8dRITru5PkbIpXpnew}{10.45.2.132}{10.45.2.132:9300}{lm}{ml.machine_memory=5368709120, xpack.installed=true, zone=GREEN, ml.max_open_jobs=20}, {elk-stack-es-master-eshelter-0}{kAof-xZaQhajOSlukeNXIQ}{Ag0JSqPVShuWBm9NGZG32w}{10.45.3.175}{10.45.3.175:9300}{lm}{ml.machine_memory=5368709120, ml.max_open_jobs=20, xpack.installed=true, zone=ESHELTER}, {elk-stack-es-master-green-1}{qWmtII20SG6UJAl1QANEPQ}{OD1rwvA7TCmQZNvX_gXwcg}{10.45.2.135}{10.45.2.135:9300}{lm}{ml.machine_memory=5368709120, ml.max_open_jobs=20, xpack.installed=true, zone=GREEN}, {elk-stack-es-master-eshelter-1}{wGX7DppyQc2G30WlpuKQAw}{bu3C-bKlTI6c4LgPfXROCw}{10.45.4.207}{10.45.4.207:9300}{lm}{ml.machine_memory=5368709120, ml.max_open_jobs=20, xpack.installed=true, zone=ESHELTER}]; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, 10.45.2.135:9300, 10.45.3.175:9300, 10.45.4.207:9300] from hosts providers and [{elk-stack-es-master-green-0}{TWEH9Px6TBm92a9dL-mRgg}{pNPr8dRITru5PkbIpXpnew}{10.45.2.132}{10.45.2.132:9300}{lm}{ml.machine_memory=5368709120, xpack.installed=true, zone=GREEN, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
What did you expect to see?
Elasticsearch working.
What did you see instead? Under which circumstances?
Elasticsearch not working. Errors in logs suggest it can't elect master.
Environment
- ECK version:
docker.elastic.co/eck/eck-operator:1.0.0
-
Kubernetes information:
- On premise: Yes, manually installed with kubeadm. Version 1.16.
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-13T18:08:14Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:18:29Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}