I am running a two node ES cluster and a single Kibana instance using Docker Swarm,
everything worked fine before enabling SSL/TLS on the ES nodes.
I am using same certificate without hostname verification for all the ES nodes and everything works perfect for the ES nodes. The only problem is that Kibana is not able to connect to ES cluster.
I am using the same certificate for all the ES instances and for Kibana too.
Following is my docker-compose.yml-
version: "3.2"
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
restart: always
environment:
- node.name={{.Node.Hostname}}
- discovery.seed_hosts=elasticsearch
- cluster.name=docker-cluster
- cluster.initial_master_nodes=node1,node2
- network.host=0.0.0.0
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms3g -Xmx3g"
- "ELASTIC_PASSWORD=myespassword"
- xpack.license.self_generated.type=basic
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=$CERTS_DIR/elasticsearch/elasticsearch.key
- xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
- xpack.security.http.ssl.certificate=$CERTS_DIR/elasticsearch/elasticsearch.crt
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
- xpack.security.transport.ssl.certificate=$CERTS_DIR/elasticsearch/elasticsearch.crt
- xpack.security.transport.ssl.key=$CERTS_DIR/elasticsearch/elasticsearch.key
secrets:
- source: es_ssl_key
target: $CERTS_DIR/elasticsearch/elasticsearch.key
- source: es_ssl_ca
target: $CERTS_DIR/ca/ca.crt
- source: es_ssl_crt
target: $CERTS_DIR/elasticsearch/elasticsearch.crt
ulimits:
memlock:
soft: -1
hard: -1
deploy:
mode: replicated
replicas: 1
ports:
- "9200:9200"
- "9300:9300"
volumes:
- es_data:/usr/share/elasticsearch/data
kibana:
image: docker.elastic.co/kibana/kibana:7.2.0
restart: always
environment:
- "SERVER_NAME=kibana"
- "ELASTICSEARCH_HOSTS=https://elasticsearch:9200"
- "XPACK_SECURITY_ENABLED=true"
- "ELASTICSEARCH_USERNAME=kibana"
- "ELASTICSEARCH_PASSWORD=mykbpassword"
- elasticsearch.ssl.certificateAuthorities=$CERTS_DIR/ca/ca.crt
- elasticsearch.ssl.verificationMode=certificate
secrets:
- source: es_ssl_ca
target: $CERTS_DIR/ca/ca.crt
ports:
- "5601:5601"
depends_on:
- elasticsearch
secrets:
es_ssl_key:
file: ./es_certs/elasticsearch/elasticsearch.key
es_ssl_ca:
file: ./es_certs/ca/ca.crt
es_ssl_crt:
file: ./es_certs/elasticsearch/elasticsearch.crt
volumes:
es_data:
I am using same certificate for all the ES nodes and Kibana generated using the following
create-certs.yml-
version: "2.2"
services:
create_certs:
container_name: create_certs
image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
command: >
bash -c '
yum install -y -q -e 0 unzip;
if [[ ! -f /certs/bundle.zip ]]; then
bin/elasticsearch-certutil cert --silent --pem --in config/certificates/instances.yml -out /certs/bundle.zip;
unzip /certs/bundle.zip -d /certs;
fi;
chown -R 1000:0 /certs '
user: "0"
working_dir: /usr/share/elasticsearch
volumes:
- ./es_certs:/certs
- .:/usr/share/elasticsearch/config/certificates
instances.yml
instances:
- name: elasticsearch
dns:
- elasticsearch
- localhost
ip:
- 127.0.0.1
I am able to successfully perform a curl call from inside of the Kibana container to the https Elasticsearch endpoint using the following command-
curl --cacert ca.crt -u kibana:mykbpassword https://elasticsearch:9200
However normally Kibana cannot connect to the ES nodes and I receive the following logs-
es_stack_kibana.1.bl9aozgf1b0z@node1 | {"type":"log","@timestamp":"2019-08-07T09:46:12Z","tags":["error","elasticsearch","admin"],"pid":1,"message":"Request error, retrying\nGET https://elasticsearch:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => connect ECONNREFUSED 10.0.3.2:9200"}
es_stack_kibana.1.bl9aozgf1b0z@node1 | {"type":"log","@timestamp":"2019-08-07T09:46:12Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: https://elasticsearch:9200/"}
es_stack_kibana.1.bl9aozgf1b0z@node1 | {"type":"log","@timestamp":"2019-08-07T09:46:12Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
es_stack_kibana.1.bl9aozgf1b0z@node1 | {"type":"log","@timestamp":"2019-08-07T09:46:12Z","tags":["status","plugin:xpack_main@7.2.0","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - No Living connections","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
es_stack_kibana.1.bl9aozgf1b0z@node1 | {"type":"log","@timestamp":"2019-08-07T09:46:12Z","tags":["status","plugin:graph@7.2.0","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - No Living connections","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
es_stack_kibana.1.bl9aozgf1b0z@node1 | {"type":"log","@timestamp":"2019-08-07T09:46:12Z","tags":["status","plugin:elasticsearch@7.2.0","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - No Living connections","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
es_stack_kibana.1.bl9aozgf1b0z@node1 | {"type":"log","@timestamp":"2019-08-07T09:46:12Z","tags":["error","elasticsearch","data"],"pid":1,"message":"Request error, retrying\nGET https://elasticsearch:9200/_xpack => connect ECONNREFUSED 10.0.3.2:9200"}
es_stack_kibana.1.bl9aozgf1b0z@node1 | {"type":"log","@timestamp":"2019-08-07T09:46:12Z","tags":["warning","elasticsearch","data"],"pid":1,"message":"Unable to revive connection: https://elasticsearch:9200/"}
es_stack_kibana.1.bl9aozgf1b0z@node1 | {"type":"log","@timestamp":"2019-08-07T09:46:12Z","tags":["warning","elasticsearch","data"],"pid":1,"message":"No living connections"}
