Hi Team,
I have configued elasticsearch statefulset object using official elasticsearch helm chart https://github.com/elastic/helm-charts. I enabeld xpack security feature and created a CA certificate on one of the elasticsearch master node using bin/elasticsearch-certutil ca command and then generated a certificate and private key on one need using bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 command. it generate one file called elastic-certificates.p12.
After enabled SSL/TLS settings, my elasticsearch pods are not working properly though their status shows "Running", it is faling to pass Readyness probe. Do i have to create CA certificate on each master node or just on one single nodes enough? Do i have to create any user and set password in k8s? Why I am asking is, according this link https://github.com/elastic/helm-charts/blob/master/elasticsearch/examples/security/security.yml, a username and password being set and exported as env.
I created a secret object using those CA and key kubectl create secret generic elastic-certificates --from-file=/root/certs/ -n logging
[root@cesium-kibana1 certs]# kubectl describe secret elastic-certificates -n logging
Name: elastic-certificates
Namespace: logging
Labels: <none>
Annotations: <none>
Type: Opaque
Data
====
elastic-certificates.p12: 2527 bytes
elastic-stack-ca.p12: 2527 bytes
I updated values.yaml file with below given entries.
elasticsearch.yml: |
elasticsearch.yml: |
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
I deployed the elasticsearch statefulset object using helm chart
helm install --name elasticsearch --tiller-namespace=logging --namespace logging elasticsearch/
These secrets are visible inside elasticsearch nodes unde /usr/share/elasticsearch/config/certs folder.
sh-4.2$ pwd
/usr/share/elasticsearch/config/certs
sh-4.2$ ls
elastic-certificates.p12 elastic-stack-ca.p12
sh-4.2$
Here are the outputs from my cluster
Normal Created 2m39s kubelet, cesium-kibana4.cisco.com Created container elasticsearch
Normal Started 2m39s kubelet, cesium-kibana4.cisco.com Started container elasticsearch
Warning Unhealthy 3s (x15 over 2m23s) kubelet, cesium-kibana4.cisco.com Readiness probe failed: Waiting fosearch cluster to become ready (request params: "wait_for_status=green&timeout=1s" )
Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )
[root@cesium-kibana1 ~]#
Output entries from elasticsearch.yaml file
Stacktrace from pod
"stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-1][10.233.108.58:9300][internal:cluster/coordination/join]",
"Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: became follower",
"at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.lambda$close$3(JoinHelper.java:467) [elasticsearch-7.5.2.jar:7.5.2]",
"at java.util.HashMap$Values.forEach(HashMap.java:981) [?:?]",
"at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.close(JoinHelper.java:467) [elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.cluster.coordination.Coordinator.becomeFollower(Coordinator.java:613) [elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.cluster.coordination.Coordinator.onFollowerCheckRequest(Coordinator.java:251) [elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.cluster.coordination.FollowersChecker$2.doRun(FollowersChecker.java:188) [elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.2.jar:7.5.2]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
"at java.lang.Thread.run(Thread.java:830) [?:?]"] }
{"type": "server", "timestamp": "2020-02-07T10:37:01,361Z", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "failed to join {elasticsearch-master-0}{OVuzaqdCQQ-PSK6eqR_5kw}{Gyad142iS52wZdfUHJnwrw}{10.233.106.53}{10.233.106.53:9300}{dilm}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-1}{NGBVof5BQI2myGsx_OnnBg}{3j8Wwbq0S5C0eZZOdQPKdA}{10.233.108.58}{10.233.108.58:9300}{dilm}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=17, lastAcceptedTerm=14, lastAcceptedVersion=271, sourceNode={elasticsearch-master-1}{NGBVof5BQI2myGsx_OnnBg}{3j8Wwbq0S5C0eZZOdQPKdA}{10.233.108.58}{10.233.108.58:9300}{dilm}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{OVuzaqdCQQ-PSK6eqR_5kw}{Gyad142iS52wZdfUHJnwrw}{10.233.106.53}{10.233.106.53:9300}{dilm}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}}]}",
"stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][10.233.106.53:9300][internal:cluster/coordination/join]",
"Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed",
"at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$4.onFailure(Coordinator.java:1429) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:88) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:68) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1349) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:125) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:173) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.cluster.coordination.Publication.access$500(Publication.java:42) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.cluster.coordination.Publication$PublicationTarget$PublishResponseHandler.onFailure(Publication.java:369) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.cluster.coordination.Coordinator$5.onFailure(Coordinator.java:1117) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.cluster.coordination.PublicationTransportHandler.lambda$sendClusterStateToNode$6(PublicationTransportHandler.java:271) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.cluster.coordination.PublicationTransportHandler$3.handleException(PublicationTransportHandler.java:289) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1120) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1120) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:243) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
"at java.lang.Thread.run(Thread.java:830) [?:?]",
"Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: non-failed nodes do not form a quorum",
"at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:171) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.cluster.coordination.Publication.access$500(Publication.java:42) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.cluster.coordination.Publication$PublicationTarget$PublishResponseHandler.onFailure(Publication.java:369) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.cluster.coordination.Coordinator$5.onFailure(Coordinator.java:1117) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.cluster.coordination.PublicationTransportHandler.lambda$sendClusterStateToNode$6(PublicationTransportHandler.java:271) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.cluster.coordination.PublicationTransportHandler$3.handleException(PublicationTransportHandler.java:289) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1120) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1120) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:243) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) ~[elasticsearch-7.5.2.jar:7.5.2]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
"at java.lang.Thread.run(Thread.java:830) ~[?:?]"] }