So, I am trying to set up the following configuration:
3 node elastic cluster
1 node logstash
1 node kibana
I am utilizing the Helm Charts (GitHub - elastic/helm-charts: You know, for Kubernetes); and specifically I have referred to the "Security" examples for ElasticSearch , Kibana, and Logstash. I am utilizing my own TLS certs, as I want them to use existing TLS certificates I use for other things, such as TLS-enabled ingress for other services in the cluster. I am deploying into an AKS K8s cluster; and I am using a basic license.
However, when I have everything turned on, my Elastic cluster NEVER goes healthy. The logs make it seem like it should be happy, but it's not.
Here is my values.yaml file for Elasticsearch:
clusterName: "data-elk"
nodeGroup: "master"
roles:
master: "true"
ingest: "true"
data: "true"
protocol: https
esJavaOpts: "-Xmx1g -Xms1g"
image: "<internal-registry>/docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "7.14.2"
imagePullSecrets:
- name: intregcred
esConfig:
elasticsearch.yml : |
xpack.license.self_generated.type: basic
xpack.security.enabled: true
# Transport SSL
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/certs/tls.key
xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/certs/tls.crt
xpack.security.transport.ssl.certificate_authorities: /usr/share/elasticsearch/config/certs/ca.crt
xpack.security.transport.ssl.verification_mode: certificate
# Http SSL
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /usr/share/elasticsearch/config/certs/tls.key
xpack.security.http.ssl.certificate: /usr/share/elasticsearch/config/certs/tls.crt
xpack.security.http.ssl.certificate_authorities: /usr/share/elasticsearch/config/certs/ca.crt
#Disable the GeoIP Update (no license so it won't work)
ingest.geoip.downloader.enabled: false
#Turn on DEBUG logging for Discovery so we can figure out wtf
logger.org.elasticsearch.discovery: DEBUG
logger.org.elasticsearch.action: DEBUG
logger.org.elasticsearch.broadcast: DEBUG
logger.org.elasticsearch.cluster: DEBUG
logger.org.elasticsearch.document: DEBUG
logger.org.elasticsearch.env: DEBUG
logger.org.elasticsearch.gateway: DEBUG
logger.org.elasticsearch.recovery: DEBUG
logger.org.elasticsearch.deprecation: ERROR
secretMounts:
- name: elastic-certificates
secretName: elastic-certificates
path: /usr/share/elasticsearch/config/certs
extraEnvs:
- name: ELASTIC_PASSWORD
valueFrom:
secretKeyRef:
name: elastic-credentials
key: password
- name: ELASTIC_USERNAME
valueFrom:
secretKeyRef:
name: elastic-credentials
key: username
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/ingress.allow-http: "false"
hosts:
- host: elasticsearch.<domain>
paths:
- path: /
tls:
- secretName: self-wildcard-tls-cert
hosts:
- elasticsearch.<domain>
Here is my values file for Kibana:
elasticsearchHosts: "https://data-elk-master:9200"
image: "<internalreg>/docker.elastic.co/kibana/kibana"
imageTag: "7.14.2"
imagePullSecrets:
- name: intregcred
extraEnvs:
- name: 'ELASTICSEARCH_USERNAME'
valueFrom:
secretKeyRef:
name: elastic-credentials
key: username
- name: 'ELASTICSEARCH_PASSWORD'
valueFrom:
secretKeyRef:
name: elastic-credentials
key: password
- name: 'KIBANA_ENCRYPTION_KEY'
valueFrom:
secretKeyRef:
name: kibana-credentials
key: enckey
kibanaConfig:
kibana.yml: |
server.ssl:
enabled: true
key: /usr/share/kibana/config/certs/tls.key
certificate: /usr/share/kibana/config/certs/tls.crt
xpack.security.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
elasticsearch.ssl:
certificateAuthorities: /usr/share/kibana/config/certs/ca.crt
verificationMode: certificate
telemetry.enabled: false
logging.root.level: debug
protocol: https
secretMounts:
- name: elastic-certificates
secretName: elastic-certificates
path: /usr/share/kibana/config/certs
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/ingress.allow-http: "false"
hosts:
- host: kibana.<domain>
paths:
- path: /
tls:
- secretName: self-wildcard-tls-cert
hosts:
- kibana.<domain>
And here is my values.yaml for Logstash:
image: "<internalreg>.azurecr.us/docker.elastic.co/logstash/logstash"
imageTag: "7.14.2"
imagePullSecrets:
- name: intregcred
persistence:
enabled: true
logstashConfig:
logstash.yml: |
http.host: 0.0.0.0
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: '${ELASTICSEARCH_USERNAME}'
xpack.monitoring.elasticsearch.password: '${ELASTICSEARCH_PASSWORD}'
xpack.monitoring.elasticsearch.hosts: ["https://data-elk-master:9200"]
xpack.monitoring.elasticsearch.ssl.certificate_authority: /usr/share/logstash/config/certs/ca.crt
logstashPipeline:
uptime.conf: |
input { exec { command => "uptime" interval => 30 } }
output { elasticsearch {
hosts => ["https://data-elk-master:9200"]
index => "logstash"
user => '${ELASTICSEARCH_USERNAME}'
password => '${ELASTICSEARCH_PASSWORD}'
cacert => "/usr/share/logstash/config/certs/ca.crt"
}
}
secretMounts:
- name: elastic-certificates
secretName: elastic-certificates
path: /usr/share/logstash/config/certs
extraEnvs:
- name: 'ELASTICSEARCH_USERNAME'
valueFrom:
secretKeyRef:
name: elastic-credentials
key: username
- name: 'ELASTICSEARCH_PASSWORD'
valueFrom:
secretKeyRef:
name: elastic-credentials
key: password
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/ingress.allow-http: "false"
hosts:
- host: logstash.<domain>
paths:
- path: /
servicePort: 8080
tls:
- secretName: self-wildcard-tls-cert
hosts:
- logstash.<domain>
These configurations are otherwise good, as if I remove the xpack tls security stuff and change the protocol definition to http (and change the kibana/logstash stuff too), everything works great.
Interesting things I have noticed:
- With the TLS config enabled, all the logs messages (will include the logs from the 3 nodes below) seem to be good, EXCEPT, that the leader will be repeatedly spitting out:
{"type": "server", "timestamp": "2021-10-06T22:48:05,409Z", "level": "DEBUG", "component": "o.e.c.s.MasterService", "cluster.name": "data-elk", "node.name": "data-elk-master-2", "message": "took [0s] to compute cluster state update for [maybe generate license for cluster]", "cluster.uuid": "RY55ngl2TBWr7cMIiCPH3w", "node.id": "3ixdfp4qT1CbQkutMIrE6A" }
and that message happens A LOT...like 109 times in 10 seconds a lot:
>>grep "maybe generate license for cluster" tlselastic-2-debug.log | wc -l
109
- When I log into any of the cluster nodes and attempt to GET /_license, I get:
[elasticsearch@data-elk-master-0 ~]$ curl -u admuser:<password> https://localhost:9200/_license
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number
...if I do the same thing but go to http: here's what I get:
[elasticsearch@data-elk-master-0 ~]$ curl -u admuser:<password> http://localhost:9200/_license
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [admuser] for REST request [/_license]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"unable to authenticate user [admuser] for REST request [/_license]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}
admuser is the user I created in the elastic-credentials secret (ELASTIC_USERNAME). Not sure why it can't authenticate? I'm assuming this is all related.
- When I set protocol to HTTP in the values files and change the URLs in Kibana and Logstash, Elastic goes healthy, logstash works (and I can login to ES via the ingress, use the generated username/pw, and verify that stuff exists by going to /_cat/indices), but Kibana does NOT.
I would attach logs, but apparently .txt files are not allowed? Only picture files?