Hello,
I am aware that other people have asked this before but it seems that none of those answers fit my case. I need to enable encryption on the communication between the nodes with TLS. (Currently one ES node + one Kibana). I need to achieve this using docker-compose. I am using self-signed certificates because it is a private network. The error I am getting is the following:
kib01 | {"type":"log","@timestamp":"2021-11-03T10:55:45+00:00","tags":["error","savedobjects-service"],"pid":18,"message":"Unable to retrieve version information from Elasticsearch nodes. write EPROTO 140622041552768:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:\n"
Inside of my kibana container I am trying:
bash-4.4$ curl -vv --cacert /usr/share/elasticsearch/config/certificates/ca/ca.crt -u 'xx:yy' https://es01:9200/_cat/nodes
* Uses proxy env variable https_proxy == 'http://10.169.127.8:8080'
* Trying 10.169.127.8...
* TCP_NODELAY set
* Connected to 10.169.127.8 (10.169.127.8) port 8080 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to es01:9200
* Server auth using Basic with user 'elastic'
> CONNECT es01:9200 HTTP/1.1
> Host: es01:9200
> User-Agent: curl/7.61.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /usr/share/elasticsearch/config/certificates/ca/ca.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to es01:9200
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to es01:9200
Unsetting the proxy has the same result.
In order to have the certificates, I am lunching a container that creates them and copy them into the nodes. This is how the file looks like:
version: '3'
services:
create_certs:
image: docker.elastic.co/elasticsearch/elasticsearch:7.14.1
container_name: create_certs
command: >
bash -c '
if [[ ! -f /certs/bundle.zip ]]; then
bin/elasticsearch-certutil cert --silent --pem --in config/certificates/instances.yml -out /certs/bundle.zip;
unzip /certs/bundle.zip -d /certs;
fi;
chown -R 1000:0 /certs
rm /certs/bundle.zip
'
working_dir: /usr/share/elasticsearch
volumes:
- certs:/certs
- .:/usr/share/elasticsearch/config/certificates
networks:
- elastic
volumes:
certs:
driver: local
networks:
elastic:
driver: bridge
The results of running this is that each container specified on the instances.yml file, has the following certs:
On kib01:
bash-4.4$ cd /usr/share/elasticsearch/config/certificates/
bash-4.4$ ls
ca es01 kib01
bash-4.4$ ls kib01/
kib01.crt kib01.key
The docker-compose file looks like this:
version: '3'
services:
es01:
build:
context: .
dockerfile: docker/elasticsearch/Dockerfile
container_name: es01
environment:
- node.name=es01
- cluster.name=xxx
- discovery.type=single-node
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms2g -Xmx2g"
- xpack.security.enabled=true
- ingest.geoip.downloader.enabled=false
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.security.transport.ssl.client_authentication=required
- xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certificates/ca/ca.crt
- xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certificates/es01/es01.crt
- xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certificates/es01/es01.key
labels:
co.elastic.logs/module: elasticsearch
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data01:/usr/share/elasticsearch/data
- certs:/usr/share/elasticsearch/config/certificates
ports:
- 9200:9200
networks:
- elastic
kib01:
build:
context: .
dockerfile: docker/kibana/Dockerfile
container_name: kib01
ports:
- 5601:5601
environment:
ELASTICSEARCH_URL: https://es01:9200
ELASTICSEARCH_HOSTS: '["https://es01:9200"]'
ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: /usr/share/elasticsearch/config/certificates/ca/ca.crt
SERVER_SSL_ENABLED: "true"
SERVER_SSL_KEY: /usr/share/elasticsearch/config/certificates/kib01/kib01.key
SERVER_SSL_CERTIFICATE: /usr/share/elasticsearch/config/certificates/kib01/kib01.crt
volumes:
- ./docker/kibana/kibana.yml:/usr/share/kibana/config/kibana.yml
- certs:/usr/share/elasticsearch/config/certificates
depends_on:
- "es01"
networks:
- elastic
volumes:
data01:
filebeat-data:
certs:
driver: local
networks:
elastic:
driver: bridge
The instances.yml file:
instances:
- name: es01
dns:
- es01
- localhost
ip:
- 127.0.0.1
- name: 'kib01'
dns:
- kib01
- localhost
ip:
- 127.0.0.1
I am following advice from here and here
Can anyone please tell me help me to address this issue? It is also the first time that I am dealing with certs and I am trying to grasp how to use them appropriately. Any hint is highly appreciated.
Thank you