Elastic Kibana TLS/SSL fails on Docker Swarm

Hi,

I have deployed 3 node Elastic & Kibana on a 3 Node Docker Swarm.
I could see the Cluster is up & running with 3 node elastic & 1 Kibana.

Now, I'm trying to Provide Elasticsearch & Kibana with TLS/SSL authentication.
I have generated the certs using below commands:

bin/elasticsearch-certutil ca &
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
bin/elasticsearch-certutil cert --ca config/certs/elastic-stack-ca.p12 -name "CN=something,OU=Consulting Team,DC=mydomain,DC=com"
openssl pkcs12 -in client.p12 -nocerts -nodes > client.key
openssl pkcs12 -in client.p12 -clcerts -nokeys > client.cer
openssl pkcs12 -in client.p12 -cacerts -nokeys -chain > client-ca.cer

I have copied all these certificates to my local VM & built a docker-compose file as below:

version: "3.7"
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.5.0
container_name: es01
environment:
- node.name=es01
- cluster.name=elk-cluster
- discovery.seed_hosts=es02,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=false
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- xpack.security.enabled=true
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.keystore.type=PKCS12
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.security.transport.ssl.keystore.path=/usr/share/elasticsearch/config/certs/elastic-stack-ca.p12
- xpack.security.transport.ssl.truststore.path=/usr/share/elasticsearch/config/certs/elastic-stack-ca.p12
- xpack.security.transport.ssl.truststore.type=PKCS12
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.keystore.path=/usr/share/elasticsearch/config/certs/elastic-certificates.p12
- xpack.security.http.ssl.truststore.path=/usr/share/elasticsearch/config/certs/elastic-certificates.p12
- xpack.security.http.ssl.client_authentication=optional
networks:
- dockerelk
ports:
- "9200:9200"
- "9300:9300"
volumes:
- ./elasticsearch/config/certs/elastic-stack-ca.p12:/usr/share/elasticsearch/config/certs/elastic-stack-ca.p12
- ./elasticsearch/config/certs/elastic-certificates.p12:/usr/share/elasticsearch/config/certs/elastic-certificates.p12
- x2es01:/usr/share/elasticsearch/data
extra_hosts:
- "host1:192.168.x.x"
- "host2:192.168.x.x"
- "host3:192.168.x.x"
deploy:
replicas: 1

es02:
image: docker.elastic.co/elasticsearch/elasticsearch:7.5.0
container_name: es02
environment:
- node.name=es02
- cluster.name=elk-cluster
- discovery.seed_hosts=es01,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=false
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- xpack.security.enabled=true
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.keystore.type=PKCS12
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.security.transport.ssl.keystore.path=/usr/share/elasticsearch/config/certs/elastic-stack-ca.p12
- xpack.security.transport.ssl.truststore.path=/usr/share/elasticsearch/config/certs/elastic-stack-ca.p12
- xpack.security.transport.ssl.truststore.type=PKCS12
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.keystore.path=/usr/share/elasticsearch/config/certs/elastic-certificates.p12
- xpack.security.http.ssl.truststore.path=/usr/share/elasticsearch/config/certs/elastic-certificates.p12
- xpack.security.http.ssl.client_authentication=optional
networks:
- dockerelk
ports:
- "9201:9200"
- "9301:9300"
volumes:
- ./elasticsearch/config/certs/elastic-stack-ca.p12:/usr/share/elasticsearch/config/certs/elastic-stack-ca.p12
- ./elasticsearch/config/certs/elastic-certificates.p12:/usr/share/elasticsearch/config/certs/elastic-certificates.p12
- x2es02:/usr/share/elasticsearch/data
extra_hosts:
- "host1:192.168.x.x"
- "host2:192.168.x.x"
- "host3:192.168.x.x"
deploy:
replicas: 1

es03:
image: docker.elastic.co/elasticsearch/elasticsearch:7.5.0
container_name: es03
environment:
- node.name=es03
- cluster.name=elk-cluster
- discovery.seed_hosts=es01,es02
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=false
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- xpack.security.enabled=true
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.keystore.type=PKCS12
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.security.transport.ssl.keystore.path=/usr/share/elasticsearch/config/certs/elastic-stack-ca.p12
- xpack.security.transport.ssl.truststore.path=/usr/share/elasticsearch/config/certs/elastic-stack-ca.p12
- xpack.security.transport.ssl.truststore.type=PKCS12
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.keystore.path=/usr/share/elasticsearch/config/certs/elastic-certificates.p12
- xpack.security.http.ssl.truststore.path=/usr/share/elasticsearch/config/certs/elastic-certificates.p12
- xpack.security.http.ssl.client_authentication=optional
networks:
- dockerelk
ports:
- "9202:9200"
- "9302:9300"
volumes:
- ./elasticsearch/config/certs/elastic-stack-ca.p12:/usr/share/elasticsearch/config/certs/elastic-stack-ca.p12
- ./elasticsearch/config/certs/elastic-certificates.p12:/usr/share/elasticsearch/config/certs/elastic-certificates.p12
- x2es03:/usr/share/elasticsearch/data
extra_hosts:
- "host1:192.168.x.x"
- "host2:192.168.x.x"
- "host3:192.168.x.x"
deploy:
replicas: 1

kibana01:
image: docker.elastic.co/kibana/kibana:7.5.0
container_name: kib01
environment:
- ELASTICSEARCH_URL=https://192.168.x.x:9200
- ELASTICSEARCH_HOSTS=https://192.168.x.x:9200
- xpack.security.enabled=true
- ELASTICSEARCH_USERNAME=kibana
- ELASTICSEARCH_PASSWORD=kibana
- elasticsearch.ssl.certificate=/usr/share/kibana/config/certs/client.cer
- elasticsearch.ssl.key=/usr/share/kibana/config/certs/client.key
- elasticsearch.ssl.certificateAuthorities=/usr/share/kibana/config/certs/client-ca.cer
- elasticsearch.ssl.verificationMode=certificate
networks:
- dockerelk
ports:
- "5601:5601"
volumes:
- ./kibana/config/certs/client.cer:/usr/share/kibana/config/certs/client.cer
- ./kibana/config/certs/client-ca.cer:/usr/share/kibana/config/certs/client-ca.cer
- ./kibana/config/certs/client.key:/usr/share/kibana/config/certs/client.key
- x2kibgit:/usr/share/kibana
networks:
dockerelk:
external: true

volumes:
x2es01:
x2es02:
x2es03:
x2kibgit:

Also, I have created the dockernet network prior running the compose file.

As, I was unable to set passwords using ./elasticsearch-setup-passwords interactive
So I created a custom user & password using below command.

bin/elasticsearch-users useradd elkuser -p elkpass -r superuser.

So, Now I'am able to login lo all my 3 ES instances using https://192.168.x.x:9200 with elkuser & elkpass command.
Also, I see my cluster ES is healthy Now.

But, not sure why Kibana is not up & running when using TLS/SSL connection.
In my Kibana docker container logs I see that following Warnings:

{"type":"log","@timestamp":"2020-05-22T14:37:19Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-05-22T14:37:20Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"Unable to revive connection: https://192.168.x.x:9200/"}
{"type":"log","@timestamp":"2020-05-22T14:37:20Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-05-22T14:37:20Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch for the [data] cluster. Error: No Living connections"}
{"type":"log","@timestamp":"2020-05-22T14:37:21Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://192.168.x.x:9200/"}
{"type":"log","@timestamp":"2020-05-22T14:37:21Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-05-22T14:37:21Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://192.168.x.x:9200/"}

Could someone of you help me in understanding the root cause of this issue & the meaning for these logs.. if either it's related to docker network or swarm setup or it's issue with certificates.
Any help would be appreciated.

Thanks in Advance

Please don't open multiple posts for the same issue. I think this is more complete so do you mind deleting the other one ?

yeah , Will remove that old post and could you suggest me anything i should do to get that done @ikakavas.

These do not contain the output filename that you chose so I'm making assumptions.

- elasticsearch.ssl.certificate=/usr/share/kibana/config/certs/client.cer
- elasticsearch.ssl.key=/usr/share/kibana/config/certs/client.key
- elasticsearch.ssl.certificateAuthorities=/usr/share/kibana/config/certs/client-ca.cer

This means that kibana will attempt to authetnicate itself to elasticsearch using TLS mutual authentication in addition to authenticating with a username and password

- ELASTICSEARCH_USERNAME=kibana
- ELASTICSEARCH_PASSWORD=kibana

Is this what you want or not?

If yes, then you need to enable a PKI realm in elasticsearch

If not, you need to remove

- elasticsearch.ssl.certificate=/usr/share/kibana/config/certs/client.cer
- elasticsearch.ssl.key=/usr/share/kibana/config/certs/client.key

Thank you @ikakavas , I will check and update you on this.