I'm running a one node Elastic cluster with Elasticsearch and Kibana. The configuration worked before on an earlier version about a year ago, however, after Kibanas version automatically updated due to new container images, the following error started appearing:
kubectl logs kibana-pod
{"type":"log","@timestamp":"2022-02-21T08:47:48+00:00","tags":["error","Elasticsearch-service"],"pid":7,"message":"Unable to retrieve version information from Elasticsearch nodes. unable to verify the first certificate"}
kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:25:17Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:19:12Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}
kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:24:08Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}
I use Let'sEncrypt certificates for Elasticsearch and Kibana. I also renewed the certificates to make sure the configuration has fresh ones.
I use the helm charts for both Elasticsearch and Kibana, however, I use some custom values:
kibana_values.yaml
---
elasticsearchHosts: "https://redacted"
extraEnvs:
- name: "NODE_OPTIONS"
value: "--max-old-space-size=1800"
- name: 'ELASTICSEARCH_USERNAME'
valueFrom:
secretKeyRef:
name: elastic-credentials
key: username
- name: 'ELASTICSEARCH_PASSWORD'
valueFrom:
secretKeyRef:
name: elastic-credentials
key: password
- name: 'KIBANA_ENCRYPTION_KEY'
valueFrom:
secretKeyRef:
name: kibana
key: encryptionkey
secretMounts:
- name: elastic-certificates
secretName: elastic-certificates
path: /usr/share/kibana/config/certs-gen/
kibanaConfig:
kibana.yml: |
server.ssl:
enabled: true
key: /usr/share/kibana/config/certs-gen/privkey2.pem
certificate: /usr/share/kibana/config/certs-gen/cert2.pem
xpack.reporting.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
xpack.security.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
xpack.encryptedSavedObjects.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
elasticsearch.ssl:
certificateAuthorities: /usr/share/kibana/config/certs-gen/fullchain2.pem
verificationMode: certificate
protocol: https
service:
type: NodePort
loadBalancerIP: ""
port: 5601
nodePort: 30002
labels: {}
annotations: {}
loadBalancerSourceRanges: []
httpPortName: http
These are the values for my elastic_search.yml
replicas: 1
minimumMasterNodes: 1
esConfig:
elasticsearch.yml: |
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/certs-gen/privkey2.pem
xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/certs-gen/cert2.pem
xpack.security.transport.ssl.certificate_authorities: [ "/usr/share/elasticsearch/config/certs-gen/fullchain2.pem" ]
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.key: /usr/share/elasticsearch/config/certs-gen/privkey2.pem
xpack.security.http.ssl.certificate: /usr/share/elasticsearch/config/certs-gen/cert2.pem
xpack.security.http.ssl.certificate_authorities: [ "/usr/share/elasticsearch/config/certs-gen/fullchain2.pem" ]
extraEnvs:
- name: ELASTIC_PASSWORD
valueFrom:
secretKeyRef:
name: elastic-credentials
key: password
- name: ELASTIC_USERNAME
valueFrom:
secretKeyRef:
name: elastic-credentials
key: username
secretMounts:
- name: elastic-certificates
secretName: elastic-certificates
path: /usr/share/elasticsearch/config/certs-gen/
protocol: https
service:
labels: {}
labelsHeadless: {}
type: NodePort
nodePort: 30001
annotations: {}
httpPortName: http
transportPortName: transport
loadBalancerIP: ""
loadBalancerSourceRanges: []
externalTrafficPolicy: ""
clusterHealthCheckParams: "wait_for_status=green&timeout=1s"