SSL_ERROR_SYSCALL in connection to es01:9200

Hello,

I am aware that other people have asked this before but it seems that none of those answers fit my case. I need to enable encryption on the communication between the nodes with TLS. (Currently one ES node + one Kibana). I need to achieve this using docker-compose. I am using self-signed certificates because it is a private network. The error I am getting is the following:

kib01    | {"type":"log","@timestamp":"2021-11-03T10:55:45+00:00","tags":["error","savedobjects-service"],"pid":18,"message":"Unable to retrieve version information from Elasticsearch nodes. write EPROTO 140622041552768:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:\n"

Inside of my kibana container I am trying:

bash-4.4$ curl -vv --cacert /usr/share/elasticsearch/config/certificates/ca/ca.crt -u 'xx:yy' https://es01:9200/_cat/nodes
* Uses proxy env variable https_proxy == 'http://10.169.127.8:8080'
*   Trying 10.169.127.8...
* TCP_NODELAY set
* Connected to 10.169.127.8 (10.169.127.8) port 8080 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to es01:9200
* Server auth using Basic with user 'elastic'
> CONNECT es01:9200 HTTP/1.1
> Host: es01:9200
> User-Agent: curl/7.61.1
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /usr/share/elasticsearch/config/certificates/ca/ca.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to es01:9200 
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to es01:9200 

Unsetting the proxy has the same result.

In order to have the certificates, I am lunching a container that creates them and copy them into the nodes. This is how the file looks like:

version: '3'

services:
  create_certs:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.14.1
    container_name: create_certs
    command: >
      bash -c '
        if [[ ! -f /certs/bundle.zip ]]; then
          bin/elasticsearch-certutil cert --silent --pem --in config/certificates/instances.yml -out /certs/bundle.zip;
          unzip /certs/bundle.zip -d /certs;
        fi;
        chown -R 1000:0 /certs
        rm /certs/bundle.zip
      '
    working_dir: /usr/share/elasticsearch
    volumes:
      - certs:/certs
      - .:/usr/share/elasticsearch/config/certificates

    networks:
      - elastic

volumes:
  certs:
      driver: local

networks:
  elastic:
    driver: bridge

The results of running this is that each container specified on the instances.yml file, has the following certs:

On kib01:

bash-4.4$ cd /usr/share/elasticsearch/config/certificates/
bash-4.4$ ls
ca  es01  kib01
bash-4.4$ ls kib01/
kib01.crt  kib01.key

The docker-compose file looks like this:

version: '3'

services:
  es01:
    build: 
      context: .
      dockerfile: docker/elasticsearch/Dockerfile
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=xxx
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
      - xpack.security.enabled=true
      - ingest.geoip.downloader.enabled=false
      - xpack.security.transport.ssl.enabled=true 
      - xpack.security.transport.ssl.verification_mode=certificate 
      - xpack.security.transport.ssl.client_authentication=required
      - xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certificates/ca/ca.crt
      - xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certificates/es01/es01.crt
      - xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certificates/es01/es01.key
    labels:
      co.elastic.logs/module: elasticsearch
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
      - certs:/usr/share/elasticsearch/config/certificates
    ports:
      - 9200:9200
    networks:
      - elastic

  kib01:
    build: 
      context: .
      dockerfile: docker/kibana/Dockerfile
    container_name: kib01
    ports:
      - 5601:5601
    environment:
      ELASTICSEARCH_URL: https://es01:9200
      ELASTICSEARCH_HOSTS: '["https://es01:9200"]'
      ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: /usr/share/elasticsearch/config/certificates/ca/ca.crt
      SERVER_SSL_ENABLED: "true"
      SERVER_SSL_KEY: /usr/share/elasticsearch/config/certificates/kib01/kib01.key
      SERVER_SSL_CERTIFICATE: /usr/share/elasticsearch/config/certificates/kib01/kib01.crt
    volumes:
      - ./docker/kibana/kibana.yml:/usr/share/kibana/config/kibana.yml
      - certs:/usr/share/elasticsearch/config/certificates
    depends_on: 
      - "es01"
    networks:
      - elastic

volumes:
  data01:
  filebeat-data:
  certs:
    driver: local

networks:
  elastic:
    driver: bridge

The instances.yml file:

instances:
  - name: es01
    dns:
      - es01
      - localhost
    ip:
      - 127.0.0.1
      
  - name: 'kib01'
    dns:
      - kib01
      - localhost
    ip:
      - 127.0.0.1

I am following advice from here and here

Can anyone please tell me help me to address this issue? It is also the first time that I am dealing with certs and I am trying to grasp how to use them appropriately. Any hint is highly appreciated.

Thank you

I have just realized that it seems that the certs are not available on one of the sides, which I cannot understand because I can see them on each node on the specified directory. From kib01:

[root@ffec28f6d628 kibana]# curl -vv --cacert /usr/share/elasticsearch/config/certificates/ca/ca.crt -u 'xx:yy' https://es01:9300/_cat/nodes
*   Trying 172.29.0.2...
* TCP_NODELAY set
* Connected to es01 (172.29.0.2) port 9300 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /usr/share/elasticsearch/config/certificates/ca/ca.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=es01
*  start date: Nov  3 16:05:39 2021 GMT
*  expire date: Nov  2 16:05:39 2024 GMT
*  subjectAltName: host "es01" matched cert's "es01"
*  issuer: CN=Elastic Certificate Tool Autogenerated CA
*  SSL certificate verify ok.
* Server auth using Basic with user 'elastic'
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET /_cat/nodes HTTP/1.1
> Host: es01:9300
> Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==
> User-Agent: curl/7.61.1
> Accept: */*
> 
* TLSv1.3 (IN), TLS alert, [no content] (0):
* TLSv1.3 (IN), TLS alert, bad certificate (554):
* OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0
* Closing connection 0
curl: (56) OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0
[root@ffec28f6d628 kibana]#

Still stuck with this issue...

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.