I'm trying to setup Elasticsearch and Kibana v7.2 in GKE using the elastic helm-charts. I've followed the instructions to enable security on these pages:
https://github.com/elastic/helm-charts/blob/master/elasticsearch/README.md#security
https://github.com/elastic/helm-charts/blob/master/kibana/README.md#security
I've setup the CA and generated the needed certificates to enable transport security; and my es cluster and kibana pods come up fine. But when I navigate to kibana from the browser, I get invalid username/password when I attempt to login with the 'elastic' user and password. However if I exec into the kibana pod, I can successfully curl the elasticsearch endpoint using the same username and password.
I have even tried disabling https setup to just test with transport ssl (although Kibana was successfully starting with https as well).
Note: The below setup is with https turned off
Elasticsearch setup
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/instance.p12
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/instance.p12
- elastic username and password are set using k8s secrets through the ELASTIC_USERNAME and ELASTIC_PASSWORD environment variables
- The p12 cert mentioned above is also mounted through k8s secrets
Kibana Setup
xpack.security.encryptionKey: <redacted>
- elastic username and password are set using k8s secrets through the ELASTICSEARCH_USERNAME and ELASTICSEARCH_PASSWORD environment variables
Info
Here is the curl performed from on the kibana Pod
bash-4.2$ curl -k -v http://elasticsearch-master:9200 -k -u $ELASTICSEARCH_USERNAME:$ELASTICSEARCH_PASSWORD
* About to connect() to elasticsearch-master port 9200 (#0)
* Trying 10.0.17.113...
* Connected to elasticsearch-master (10.0.17.113) port 9200 (#0)
* Server auth using Basic with user 'elastic'
> GET / HTTP/1.1
> Authorization: Basic ZWxhc3RpYzojMjFqTGhQSUde
> User-Agent: curl/7.29.0
> Host: elasticsearch-master:9200
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 517
<
{
"name" : "elasticsearch-master-1",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "B-tG7xTSR6-t-XmzL18s8w",
"version" : {
"number" : "7.2.0",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "508c38a",
"build_date" : "2019-06-20T15:54:18.811730Z",
"build_snapshot" : false,
"lucene_version" : "8.0.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
Error from elasticsearch log when failed UI login
{"type": "server", "timestamp": "2019-07-15T14:01:06,758+0000", "level": "INFO", "component": "o.e.x.s.a.AuthenticationService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "cluster.uuid": "B-tG7xTSR6-t-XmzL18s8w", "node.id": "8zbrBdHoT0mxyGrkUqp45g", "message": "Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]" }