Completed handshake with but follow up connection failed

Elastic Stack Elasticsearch
1

I'm trying to install via podman compose in 3 machines on AWS EC2

version: "3"
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.12.0
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-thaicom-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - "ES_JAVA_OPTS=-Xms3g -Xmx3g"
      - index.codec=best_compression
      - network.host=127.0.0.1
      - http.host=0.0.0.0
    extra_hosts:
      - "es01:127.0.0.1"
      - "es02:172.31.25.120"
      - "es03:172.31.18.77"
    dns:
      - 127.0.0.53
    ports:
      - 9200:9200
      - 9300:9300

But I got errors like

{
    "type": "server",
    "timestamp": "2021-04-27T09:17:10,150Z",
    "level": "WARN",
    "component": "o.e.d.HandshakingTransportAddressConnector",
    "cluster.name": "es-thaicom-cluster",
    "node.name": "es01",
    "message": "[connectToRemoteMasterNode[172.31.18.77:9300]] completed handshake with [{es03}{OJUZTeWBTNy5GBwJfzdGpA}{2g4EtF2ASICtc0h5bUv9-A}{10.0.2.100}{10.0.2.100:9300}{cdfhilmrstw}{ml.machine_memory=4124807168, ml.max_open_jobs=20, xpack.installed=true, ml.max_jvm_size=3221225472, transform.node=true}] but followup connection failed",
    "stacktrace": [
        "org.elasticsearch.transport.ConnectTransportException: [es03][10.0.2.100:9300] handshake failed. unexpected remote node {es01}{c-2xngTRTuS2TEU01hXFGA}{qzEYacQCSRq8XH9jrSdI3w}{10.0.2.100}{10.0.2.100:9300}{cdfhilmrstw}{ml.machine_memory=4124807168, ml.max_open_jobs=20, xpack.installed=true, ml.max_jvm_size=3221225472, transform.node=true}",
        "at org.elasticsearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:417) ~[elasticsearch-7.12.0.jar:7.12.0]",
        "at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:70) [elasticsearch-7.12.0.jar:7.12.0]",
        "at org.elasticsearch.transport.TransportService$5.onResponse(TransportService.java:503) [elasticsearch-7.12.0.jar:7.12.0]",
        "at org.elasticsearch.transport.TransportService$5.onResponse(TransportService.java:493) [elasticsearch-7.12.0.jar:7.12.0]",
        "at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:43) [elasticsearch-7.12.0.jar:7.12.0]",
        "at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1280) [elasticsearch-7.12.0.jar:7.12.0]",
        "at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1280) [elasticsearch-7.12.0.jar:7.12.0]",
        "at org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:291) [elasticsearch-7.12.0.jar:7.12.0]",
        "at org.elasticsearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:279) [elasticsearch-7.12.0.jar:7.12.0]",
        "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673) [elasticsearch-7.12.0.jar:7.12.0]",
        "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]",
        "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]",
        "at java.lang.Thread.run(Thread.java:832) [?:?]"
    ]
}

How do I fix it?

2

These two nodes both appear to be at 10.0.2.100:9300. That won't work, you need to give each node its own address.

2 Likes
3

I change environment

  • Remove http.host and network.host.
  • Add network.publish_host=172.31.44.179.

It's work.

4 Likes
4

Although that might appear to work for now, I don't think it's the correct solution. See the docs on network settings, particularly for network.publish_host:

Use this setting only if binding to multiple addresses or using different addresses for publishing and binding.

I don't think either of these apply to you, so you should just set network.host.

2 Likes
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.