SSL(curl) connection error in ElasticSearch setup

Have setup a 3-node Elasticsearch cluster using docker-compose. Followed below steps: On one of the master nodes, es11, gets below error, however same curl command works fine on other 2 nodes i.e. es12, es13:

Error:

curl -X GET 'https://localhost:9316'
    curl: (35) Encountered end of file

Below error in logs:

"stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [es13][SOMEIP:9316][internal:cluster/coordination/join]",
"Caused by: org.elasticsearch.transport.ConnectTransportException: [es11][SOMEIP:9316] handshake failed. unexpected remote node {es13}{SOMEVALUE}{SOMEVALUE
"at org.elasticsearch.transport.TransportService.lambda$connectionValidator$6(TransportService.java:468) ~[elasticsearch-7.17.6.jar:7.17.6]",
"at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:95) ~[elasticsearch-7.17.6.jar:7.17.6]",
"at org.elasticsearch.transport.TransportService.lambda$handshake$9(TransportService.java:5

https://localhost:9316 on browser gives site can't be reached error as well.It seems SSL certificate as created in step 4 below is having some issues in es11. Any leads please? OR If I repeat step 4, do i need to copy the certs again to es12 & es13?

Below elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0

Ports as defined in all 3 nodes docker-compose.yml

 environment:
      - node.name=es11
      - transport.port=9316
 ports:
      - 9216:9200
      - 9316:9316
  1. Initialize a docker swarm. On ES11 run docker swarm init. Follow the instructions to join 12 and 13 to the swarm.
  2. Create an overlay network docker network create -d overlay --attachable elastic
  3. If necessary, bring down the current cluster and remove all the associated volumes by running docker-compose down -v
  4. Create SSL certificates for ES with docker-compose -f create-certs.yml run --rm create_certs
  5. Copy the certs for es12 and 13 to the respective servers
  6. Use this busybox to create the overlay network on 12 and 13 sudo docker run -itd --name containerX --net [network name] busybox
  7. Configure certs on 12 and 13 with docker-compose -f config-certs.yml run --rm config_certs
  8. Start the cluster with docker-compose up -d on each server
  9. Set the passwords for the built-in ES accounts by logging into the cluster docker exec -it es11 sh then running bin/elasticsearch-setup-passwords interactive --url localhost:9316

There's no HTTP API on the transport port, which is what you have set at 9316, use 9200 instead.

@warkolm
I'm taking reference from other environments successfully setup and working fine.Below is the docker-compose.yml file from one such env.

version: '2.2'

services:
  es11:
    image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
    container_name: es11
    environment:
      - node.name=es11
      - transport.port=9316
      - cluster.name=uat-lands
      - discovery.seed_hosts=es12,es13
      - cluster.initial_master_nodes=es11,es12,es13
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms4096m -Xmx4096m"
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.http.ssl.keystore.path=${CERTS_DIR}/es11/es11.p12
      - xpack.security.http.ssl.truststore.path=${CERTS_DIR}/es11/es11.p12
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.keystore.path=${CERTS_DIR}/es11/es11.p12
      - xpack.security.transport.ssl.truststore.path=${CERTS_DIR}/es11/es11.p12
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data:/usr/share/elasticsearch/data
      - config:/usr/share/elasticsearch/config
      - log:/usr/share/elasticsearch/logs
    ports:
      - 9216:9200
      - 9316:9316
    networks:
      - elastic
    dns:
      - es11

    healthcheck:
      test: curl -k -s https://localhost:9200 >/dev/null; if [[ $$? == 52 ]]; then echo 0; else echo 1; fi
      interval: 30s
      timeout: 10s
      retries: 5

volumes:
  data:
    driver: local
    driver_opts:
  config:
    driver: local
    driver_opts:
      type: none
      device: '/mnt/elasticmount/es11/config'
      o: bind
  log:
    driver: local
    driver_opts:
      type: none
      device: '/mnt/elasticmount/es11/log'
      o: bind

networks:
  elastic:
    driver: overlay

You cannot talk HTTP to this transport.port=9316, it's binary only and curl will never work.

You need to use port 9200 in the container, or 9216 outside the container as that is the HTTP API port.

Recieves below error when trying to use 9200 or 9216 on http/s

sudo curl -X GET 'https://localhost:9200'
curl: (7) Failed connect to localhost:9200; Connection refused

sudo curl -X GET 'https://localhost:9216'
curl: (35) Encountered end of file

Error when trying to set password with 9200 is mentioned here: Master_not_discovered_exception error while setting up passwords for user

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.