Hi all!
Having a bit of a struggle with getting Kibana talking to my elastic http service in ECK.
I'm running on an Azure cluster.
Kibana pod won't start, produces the following logs:
{"type":"log","@timestamp":"2020-06-02T15:09:13Z","tags":["error","elasticsearch","admin"],"pid":6,"message":"Request error, retrying\nGET https://elasticsearch-es-http.default.svc:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => certificate has expired"}
{"type":"log","@timestamp":"2020-06-02T15:09:13Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-es-http.default.svc:9200/"}
{"type":"log","@timestamp":"2020-06-02T15:09:13Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-06-02T15:09:13Z","tags":["error","savedobjects-service"],"pid":6,"message":"Unable to retrieve version information from Elasticsearch nodes."}
{"type":"log","@timestamp":"2020-06-02T15:09:14Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-es-http.default.svc:9200/"}
The certificate is in fact not invalid. I know this because this self same certificate is installed on the elastic box. Config as follows:
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: elasticsearch
spec:
version: 7.7.0
http:
service:
spec:
type: LoadBalancer
tls:
certificate:
secretName: elasticsearch-es-cert
nodeSets:
- name: masters
count: 3
config: # Any setting that would go into your elasticsearch.yaml goes here
node.master: true
node.data: false
node.ingest: false
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 30Gi
storageClassName: standard
podTemplate:
spec:
initContainers:
- name: sysctl
securityContext:
privileged: true
command:
["sh", "-c", "sysctl -w vm.max_map_count=262144"]
- name: coord
count: 1
config: # Any setting that would go into your elasticsearch.yaml goes here
node.master: false
node.data: false
node.ingest: false
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 30Gi
storageClassName: standard
podTemplate:
spec:
initContainers:
- name: sysctl
securityContext:
privileged: true
command:
["sh", "-c", "sysctl -w vm.max_map_count=262144"]
- name: data
count: 3
config: # Any setting that would go into your elasticsearch.yaml goes here
node.master: false
node.data: true
node.ingest: true
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 512Gi
storageClassName: premium
podTemplate:
spec:
initContainers:
- name: sysctl
securityContext:
privileged: true
command:
["sh", "-c", "sysctl -w vm.max_map_count=262144"]
containers:
- name: elasticsearch
env:
- name: ES_JAVA_OPTS
value: -Xms2g -Xmx2g
resources:
requests:
memory: 2Gi
cpu: 0.5
limits:
memory: 4Gi
cpu: 1.5
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
elasticsearch.k8s.elastic.co/cluster-name: elasticsearch
topologyKey: kubernetes.io/hostname
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: elasticsearch
spec:
version: 7.7.0
http:
service:
spec:
type: LoadBalancer
tls:
certificate:
secretName: elasticsearch-es-cert
count: 1
elasticsearchRef:
name: elasticsearch
namespace: default
I can connect to the elasticsearch cluster on it's loadbalancer's public IP address, SSL cert is valid.
If I remove the tls config from the elasticsearch cluster, Kibana fires right back up and we're off to the races.
Am I doing something wrong with how I'm configuring the tls on Elastic or Kibana?
Thanks!
