SSL(curl) connection error in ElasticSearch setup

Elastic Stack Elasticsearch
1

Have setup a 3-node Elasticsearch cluster using docker-compose. Followed below steps: On one of the master nodes, es11, gets below error, however same curl command works fine on other 2 nodes i.e. es12, es13:

Error:

curl -X GET 'https://localhost:9316'
    curl: (35) Encountered end of file

Below error in logs:

"stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [es13][SOMEIP:9316][internal:cluster/coordination/join]",
"Caused by: org.elasticsearch.transport.ConnectTransportException: [es11][SOMEIP:9316] handshake failed. unexpected remote node {es13}{SOMEVALUE}{SOMEVALUE
"at org.elasticsearch.transport.TransportService.lambda$connectionValidator$6(TransportService.java:468) ~[elasticsearch-7.17.6.jar:7.17.6]",
"at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:95) ~[elasticsearch-7.17.6.jar:7.17.6]",
"at org.elasticsearch.transport.TransportService.lambda$handshake$9(TransportService.java:5

https://localhost:9316 on browser gives site can't be reached error as well.It seems SSL certificate as created in step 4 below is having some issues in es11. Any leads please? OR If I repeat step 4, do i need to copy the certs again to es12 & es13?

Below elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0

Ports as defined in all 3 nodes docker-compose.yml

 environment:
      - node.name=es11
      - transport.port=9316
 ports:
      - 9216:9200
      - 9316:9316
  1. Initialize a docker swarm. On ES11 run docker swarm init. Follow the instructions to join 12 and 13 to the swarm.
  2. Create an overlay network docker network create -d overlay --attachable elastic
  3. If necessary, bring down the current cluster and remove all the associated volumes by running docker-compose down -v
  4. Create SSL certificates for ES with docker-compose -f create-certs.yml run --rm create_certs
  5. Copy the certs for es12 and 13 to the respective servers
  6. Use this busybox to create the overlay network on 12 and 13 sudo docker run -itd --name containerX --net [network name] busybox
  7. Configure certs on 12 and 13 with docker-compose -f config-certs.yml run --rm config_certs
  8. Start the cluster with docker-compose up -d on each server
  9. Set the passwords for the built-in ES accounts by logging into the cluster docker exec -it es11 sh then running bin/elasticsearch-setup-passwords interactive --url localhost:9316
2

There's no HTTP API on the transport port, which is what you have set at 9316, use 9200 instead.

3

@warkolm
I'm taking reference from other environments successfully setup and working fine.Below is the docker-compose.yml file from one such env.

version: '2.2'

services:
  es11:
    image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
    container_name: es11
    environment:
      - node.name=es11
      - transport.port=9316
      - cluster.name=uat-lands
      - discovery.seed_hosts=es12,es13
      - cluster.initial_master_nodes=es11,es12,es13
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms4096m -Xmx4096m"
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.http.ssl.keystore.path=${CERTS_DIR}/es11/es11.p12
      - xpack.security.http.ssl.truststore.path=${CERTS_DIR}/es11/es11.p12
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.keystore.path=${CERTS_DIR}/es11/es11.p12
      - xpack.security.transport.ssl.truststore.path=${CERTS_DIR}/es11/es11.p12
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data:/usr/share/elasticsearch/data
      - config:/usr/share/elasticsearch/config
      - log:/usr/share/elasticsearch/logs
    ports:
      - 9216:9200
      - 9316:9316
    networks:
      - elastic
    dns:
      - es11

    healthcheck:
      test: curl -k -s https://localhost:9200 >/dev/null; if [[ $$? == 52 ]]; then echo 0; else echo 1; fi
      interval: 30s
      timeout: 10s
      retries: 5

volumes:
  data:
    driver: local
    driver_opts:
  config:
    driver: local
    driver_opts:
      type: none
      device: '/mnt/elasticmount/es11/config'
      o: bind
  log:
    driver: local
    driver_opts:
      type: none
      device: '/mnt/elasticmount/es11/log'
      o: bind

networks:
  elastic:
    driver: overlay
4

You cannot talk HTTP to this transport.port=9316, it's binary only and curl will never work.

You need to use port 9200 in the container, or 9216 outside the container as that is the HTTP API port.

5

Recieves below error when trying to use 9200 or 9216 on http/s

sudo curl -X GET 'https://localhost:9200'
curl: (7) Failed connect to localhost:9200; Connection refused

sudo curl -X GET 'https://localhost:9216'
curl: (35) Encountered end of file

Error when trying to set password with 9200 is mentioned here: Master_not_discovered_exception error while setting up passwords for user

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.