Hi Team,
I have configued elasticsearch statefulset object using official elasticsearch helm chart https://github.com/elastic/helm-charts. I enabeld xpack security feature and created a CA certificate on one of the elasticsearch master node using bin/elasticsearch-certutil ca command and then generated a certificate and private key on one need using bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 command. it generate one file called elastic-certificates.p12.
After enabled SSL/TLS settings, my elasticsearch pods are not working properly though their status shows "Running", it is faling to pass Readyness probe. Do i have to create CA certificate on each master node or just on one single nodes enough? Do i have to create any user and set password in k8s? Why I am asking is, according this link https://github.com/elastic/helm-charts/blob/master/elasticsearch/examples/security/security.yml, a username and password being set and exported as env.
I created a secret object using those CA and key kubectl create secret generic elastic-certificates --from-file=/root/certs/ -n logging
[root@cesium-kibana1 certs]# kubectl describe secret elastic-certificates -n logging
Name: elastic-certificates
Namespace: logging
Labels: <none>
Annotations: <none>
Type: Opaque
Data
====
elastic-certificates.p12: 2527 bytes
elastic-stack-ca.p12: 2527 bytes
I updated values.yaml file with below given entries.
elasticsearch.yml: |
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
I deployed the elasticsearch statefulset object using helm chart
[root@kubespray helm-charts-master]# helm install --values elasticsearch/examples/security/security.yml --values elasticsearch/values.yaml elasticsearch --tiller-namespace logging --namespace logging
These secrets are visible inside elasticsearch nodes unde /usr/share/elasticsearch/config/certs folder.
sh-4.2$ pwd
/usr/share/elasticsearch/config/certs
sh-4.2$ ls
elastic-certificates.p12 elastic-stack-ca.p12
sh-4.2$
Here are the outputs from my cluster
Normal Created 2m39s kubelet, cesium-kibana4.cisco.com Created container elasticsearch
Normal Started 2m39s kubelet, cesium-kibana4.cisco.com Started container elasticsearch
Warning Unhealthy 3s (x15 over 2m23s) kubelet, cesium-kibana4.cisco.com Readiness probe failed: Waiting fosearch cluster to become ready (request params: "wait_for_status=green&timeout=1s" )
Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )
[root@cesium-kibana1 ~]#
Log messages from pod
{"type": "server", "timestamp": "2020-02-07T11:35:03,371Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:54938}", "cluster.uuid": "PmSRDy-8T0-H1CfSXAgsiw", "node.id": "NGBVof5BQI2myGsx_OnnBg" }
{"type": "server", "timestamp": "2020-02-07T11:35:13,379Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:55066}", "cluster.uuid": "PmSRDy-8T0-H1CfSXAgsiw", "node.id": "NGBVof5BQI2myGsx_OnnBg" }
{"type": "server", "timestamp": "2020-02-07T11:35:23,383Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:55174}", "cluster.uuid": "PmSRDy-8T0-H1CfSXAgsiw", "node.id": "NGBVof5BQI2myGsx_OnnBg" }
{"type": "server", "timestamp": "2020-02-07T11:35:33,392Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:55290}", "cluster.uuid": "PmSRDy-8T0-H1CfSXAgsiw", "node.id": "NGBVof5BQI2myGsx_OnnBg" }
{"type": "server", "timestamp": "2020-02-07T11:35:43,385Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:55408}", "cluster.uuid": "PmSRDy-8T0-H1CfSXAgsiw", "node.id": "NGBVof5BQI2myGsx_OnnBg" }
{"type": "server", "timestamp": "2020-02-07T11:35:53,371Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:55526}", "cluster.uuid": "PmSRDy-8T0-H1CfSXAgsiw", "node.id": "NGBVof5BQI2myGsx_OnnBg" }
{"type": "server", "timestamp": "2020-02-07T11:36:03,378Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:55642}", "cluster.uuid": "PmSRDy-8T0-H1CfSXAgsiw", "node.id": "NGBVof5BQI2myGsx_OnnBg" }
{"type": "server", "timestamp": "2020-02-07T11:36:13,377Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:55766}", "cluster.uuid": "PmSRDy-8T0-H1CfSXAgsiw", "node.id": "NGBVof5BQI2myGsx_OnnBg" }
{"type": "server", "timestamp": "2020-02-07T11:36:23,376Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:55870}", "cluster.uuid": "PmSRDy-8T0-H1CfSXAgsiw", "node.id": "NGBVof5BQI2myGsx_OnnBg" }
{"type": "server", "timestamp": "2020-02-07T11:36:33,360Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:55976}", "cluster.uuid": "PmSRDy-8T0-H1CfSXAgsiw", "node.id": "NGBVof5BQI2myGsx_OnnBg" }
{"type": "server", "timestamp": "2020-02-07T11:36:43,379Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:56126}", "cluster.uuid": "PmSRDy-8T0-H1CfSXAgsiw", "node.id": "NGBVof5BQI2myGsx_OnnBg" }
{"type": "server", "timestamp": "2020-02-07T11:36:53,374Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:56246}", "cluster.uuid": "PmSRDy-8T0-H1CfSXAgsiw", "node.id": "NGBVof5BQI2myGsx_OnnBg" }
{"type": "server", "timestamp": "2020-02-07T11:37:03,365Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:56348}", "cluster.uuid": "PmSRDy-8T0-H1CfSXAgsiw", "node.id": "NGBVof5BQI2myGsx_OnnBg" }
{"type": "server", "timestamp": "2020-02-07T11:37:13,377Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:56466}", "cluster.uuid": "PmSRDy-8T0-H1CfSXAgsiw", "node.id": "NGBVof5BQI2myGsx_OnnBg" }
{"type": "server", "timestamp": "2020-02-07T11:37:23,369Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:56588}", "cluster.uuid": "PmSRDy-8T0-H1CfSXAgsiw", "node.id": "NGBVof5BQI2myGsx_OnnBg" }
{"type": "server", "timestamp": "2020-02-07T11:37:33,374Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:56700}", "cluster.uuid": "PmSRDy-8T0-H1CfSXAgsiw", "node.id": "NGBVof5BQI2myGsx_OnnBg" }
{"type": "server", "timestamp": "2020-02-07T11:37:43,372Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:56834}", "cluster.uuid": "PmSRDy-8T0-H1CfSXAgsiw", "node.id": "NGBVof5BQI2myGsx_OnnBg" }
{"type": "server", "timestamp": "2020-02-07T11:37:53,370Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:56940}", "cluster.uuid": "PmSRDy-8T0-H1CfSXAgsiw", "node.id": "NGBVof5BQI2myGsx_OnnBg" }
{"type": "server", "timestamp": "2020-02-07T11:38:03,369Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:57050}", "cluster.uuid": "PmSRDy-8T0-H1CfSXAgsiw", "node.id": "NGBVof5BQI2myGsx_OnnBg" }
{"type": "server", "timestamp": "2020-02-07T11:38:13,374Z", "level": "WARN", "component": "o.e.x.s.t.n.SecurityNetty4HttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:57176}", "cluster.uuid": "PmSRDy-8T0-H1CfSXAgsiw", "node.id": "NGBVof5BQI2myGsx_OnnBg" }