Remote Cluster Connectivity failure

Elastic Stack Elasticsearch
1

Hello ,
I am trying to connect to a remote cluster which has connectivity with my server. But I am getting issue while connecting. I am sharing my elasticsearch.yml. Can anyone suggest whether configurations are correct or not? I am getting below error while connecting.
remote server ip : 240b:xxxx:1e1:xxxx:b452:2:1:0
elasticsearch.yml

    node.name: es-proxy-dedicated-master-node-02
    node.master: true
    node.data: false
    node.ingest: false
    network.bind_host: '[::]'
    network.host: '_global:ipv6_'
    network.publish_host: es-proxy-dedicated-master-node-01
    cluster.initial_master_nodes: es-proxy-dedicated-master-node-01
    discovery.seed_hosts: "[240b:xxxx:1e1:xxxx:b452:2:1:0]:31114"
    http.cors.enabled: true
    http.cors.allow-origin: "*"
    xpack.license.self_generated.type: basic
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
    node.attr.host_name: <host>
    path.data: /usr/share/elasticsearch/data
    cluster.remote.cluster_two.seeds: "[240b:xxxx:1e1:xxxx:b452:2:1:0]:31114"
    cluster.remote.cluster_two.proxy: "[240b:xxxx:1e1:xxxx:b452:2:1:0]:31114"
    logger.org.elasticsearch.transport: "ALL"

ERROR I am getting in logs

{"type": "server", "timestamp": "2020-07-13T11:16:02,692Z", "level": "WARN", "component": "o.e.t.RemoteClusterConnection", "cluster.name": "es-proxy", "node.name": "es-proxy-dedicated-master-node-01", "message": "fetching nodes from external cluster [cluster_two] failed", 
"stacktrace": ["org.elasticsearch.transport.ConnectTransportException: [][[240b:xxxx:1e1:xxxx:b452:2:1:0]:31114] invalid DiscoveryNode server_name [[240b:xxxx:1e1:xxxx:b452:2:1:0]]",
"at org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport$SecurityClientChannelInitializer.<init>(SecurityNetty4Transport.java:146) ~[?:?]",
"at org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport.getClientChannelInitializer(SecurityNetty4Transport.java:103) ~[?:?]",
"at org.elasticsearch.transport.netty4.Netty4Transport.initiateChannel(Netty4Transport.java:281) ~[?:?]",
"at org.elasticsearch.transport.netty4.Netty4Transport.initiateChannel(Netty4Transport.java:81) ~[?:?]",
"at org.elasticsearch.transport.TcpTransport.initiateConnection(TcpTransport.java:281) ~[elasticsearch-7.5.0.jar:7.5.0]",
"at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:266) ~[elasticsearch-7.5.0.jar:7.5.0]",
"at org.elasticsearch.transport.ConnectionManager.internalOpenConnection(ConnectionManager.java:245) ~[elasticsearch-7.5.0.jar:7.5.0]",
"at org.elasticsearch.transport.ConnectionManager.openConnection(ConnectionManager.java:92) ~[elasticsearch-7.5.0.jar:7.5.0]",
"at org.elasticsearch.transport.RemoteConnectionManager.openConnection(RemoteConnectionManager.java:62) ~[elasticsearch-7.5.0.jar:7.5.0]",
"at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.collectRemoteNodes(RemoteClusterConnection.java:403) [elasticsearch-7.5.0.jar:7.5.0]",
"at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.access$500(RemoteClusterConnection.java:301) [elasticsearch-7.5.0.jar:7.5.0]",
"at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler$1.doRun(RemoteClusterConnection.java:345) [elasticsearch-7.5.0.jar:7.5.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.0.jar:7.5.0]",
"at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]",
"at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.5.0.jar:7.5.0]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
"at java.lang.Thread.run(Thread.java:830) [?:?]",
"Caused by: java.lang.IllegalArgumentException: Contains non-LDH ASCII characters",
"at java.net.IDN.toASCIIInternal(IDN.java:296) ~[?:?]",
"at java.net.IDN.toASCII(IDN.java:122) ~[?:?]",
"at javax.net.ssl.SNIHostName.<init>(SNIHostName.java:99) ~[?:?]",
"at org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport$SecurityClientChannelInitializer.<init>(SecurityNetty4Transport.java:144) ~[?:?]",
"... 18 more"] }
{"type": "server", "timestamp": "2020-07-13T11:16:02,700Z", "level": "WARN", "component": "o.e.t.RemoteClusterService", "cluster.name": "es-proxy", "node.name": "es-proxy-dedicated-master-node-01", "message": "failed to update seed list for cluster: cluster_two", 
"stacktrace": ["org.elasticsearch.transport.ConnectTransportException: [][[240b:xxxx:1e1:xxxx:b452:2:1:0]:31114] invalid DiscoveryNode server_name [[240b:xxxx:1e1:xxxx:b452:2:1:0]]",
"at org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport$SecurityClientChannelInitializer.<init>(SecurityNetty4Transport.java:146) ~[?:?]",
"at org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport.getClientChannelInitializer(SecurityNetty4Transport.java:103) ~[?:?]",
"at org.elasticsearch.transport.netty4.Netty4Transport.initiateChannel(Netty4Transport.java:281) ~[?:?]",
"at org.elasticsearch.transport.netty4.Netty4Transport.initiateChannel(Netty4Transport.java:81) ~[?:?]",
"at org.elasticsearch.transport.TcpTransport.initiateConnection(TcpTransport.java:281) ~[elasticsearch-7.5.0.jar:7.5.0]",
"at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:266) ~[elasticsearch-7.5.0.jar:7.5.0]",
"at org.elasticsearch.transport.ConnectionManager.internalOpenConnection(ConnectionManager.java:245) ~[elasticsearch-7.5.0.jar:7.5.0]",
"at org.elasticsearch.transport.ConnectionManager.openConnection(ConnectionManager.java:92) ~[elasticsearch-7.5.0.jar:7.5.0]",
"at org.elasticsearch.transport.RemoteConnectionManager.openConnection(RemoteConnectionManager.java:62) ~[elasticsearch-7.5.0.jar:7.5.0]",
"at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.collectRemoteNodes(RemoteClusterConnection.java:403) [elasticsearch-7.5.0.jar:7.5.0]",
"at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.access$500(RemoteClusterConnection.java:301) [elasticsearch-7.5.0.jar:7.5.0]",
"at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler$1.doRun(RemoteClusterConnection.java:345) [elasticsearch-7.5.0.jar:7.5.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.0.jar:7.5.0]",
"at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]",
"at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.5.0.jar:7.5.0]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
"at java.lang.Thread.run(Thread.java:830) [?:?]",
"Caused by: java.lang.IllegalArgumentException: Contains non-LDH ASCII characters",
"at java.net.IDN.toASCIIInternal(IDN.java:296) ~[?:?]",
"at java.net.IDN.toASCII(IDN.java:122) ~[?:?]",
"at javax.net.ssl.SNIHostName.<init>(SNIHostName.java:99) ~[?:?]",
"at org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport$SecurityClientChannelInitializer.<init>(SecurityNetty4Transport.java:144) ~[?:?]",
"... 18 more"] } ```
2

TLS-protected cross-cluster connections use SNI when using a proxy, but your proxy's hostname (really, its bare ipv6 address) contains characters that are forbidden for SNI. Do you need a proxy? If not, remove the cluster.remote.cluster_two.proxy setting. If you do need a proxy then you'll need to give it a name which is valid for SNI.

3

Thanks @DavidTurner for your quick response !!
I have removed this proxy setting from elasticsearch.yml then to I am not able to connect to remote cluster.Now I am not getting any error only one debug log getting. Below is the updated elasticsearch.yml and log

    node.name: es-proxy-dedicated-master-node-01
    node.master: true
    node.data: false
    node.ingest: false
    network.bind_host: '[::]'
    network.host: '_global:ipv6_'
    cluster.initial_master_nodes: es-proxy-dedicated-master-node-01
    discovery.seed_hosts: "[240b:xxxx:1e1:xxxx:b452:2:1:0]"
    http.cors.enabled: true
    http.cors.allow-origin: "*"
    xpack.license.self_generated.type: basic
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
    node.attr.host_name: <hostname>
    path.data: /usr/share/elasticsearch/data
    cluster.remote.cluster_two.seeds: "[240b:xxxx:1e1:xxxx:b452:2:1:0]:31114"
    logger.org.elasticsearch.transport: "debug"

And here is the debug logs which I got

{"type": "server", "timestamp": "2020-07-13T14:00:28,936Z", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "es-proxy", "node.name": "es-proxy-dedicated-master-node-01", "message": "recovered [4] indices into cluster_state", "cluster.uuid": "7PJtUzeYQ2SWOIgkb4Vr0Q", "node.id": "SZ7uzWv1QKyxRGvI6tOv_Q"  }
{"type": "server", "timestamp": "2020-07-13T14:00:29,604Z", "level": "DEBUG", "component": "o.e.t.ConnectionManager", "cluster.name": "es-proxy", "node.name": "es-proxy-dedicated-master-node-01", "message": "connected to node [{es-proxy-data-node-01}{KXsvX7OFSr-od_1NSf3hWQ}{aUjhBpxjT0WEDQvmuaTjag}{es-proxy-data-node-01}{[<>]:9300}{dl}{ml.machine_memory=17179869184, ml.max_open_jobs=20, xpack.installed=true, host_name=<host>}]", "cluster.uuid": "7PJtUzeYQ2SWOIgkb4Vr0Q", "node.id": "SZ7uzWv1QKyxRGvI6tOv_Q"  }
{"type": "server", "timestamp": "2020-07-13T14:00:29,621Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "es-proxy", "node.name": "es-proxy-dedicated-master-node-01", "message": "node-join[{es-proxy-data-node-01}{KXsvX7OFSr-od_1NSf3hWQ}{aUjhBpxjT0WEDQvmuaTjag}{es-proxy-data-node-01}{[<>]:9300}{dl}{ml.machine_memory=17179869184, ml.max_open_jobs=20, xpack.installed=true, host_name=<host>} join existing leader], term: 10, version: 105, delta: added {{es-proxy-data-node-01}{KXsvX7OFSr-od_1NSf3hWQ}{aUjhBpxjT0WEDQvmuaTjag}{es-proxy-data-node-01}{[<]:9300}{dl}{ml.machine_memory=17179869184, ml.max_open_jobs=20, xpack.installed=true, host_name=<host>}}", "cluster.uuid": "7PJtUzeYQ2SWOIgkb4Vr0Q", "node.id": "SZ7uzWv1QKyxRGvI6tOv_Q"  }
{"type": "server", "timestamp": "2020-07-13T14:00:29,981Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "es-proxy", "node.name": "es-proxy-dedicated-master-node-01", "message": "added {{es-proxy-data-node-01}{KXsvX7OFSr-od_1NSf3hWQ}{aUjhBpxjT0WEDQvmuaTjag}{es-proxy-data-node-01}{[<>]:9300}{dl}{ml.machine_memory=17179869184, ml.max_open_jobs=20, xpack.installed=true, host_name=<host>}}, term: 10, version: 105, reason: Publication{term=10, version=105}", "cluster.uuid": "7PJtUzeYQ2SWOIgkb4Vr0Q", "node.id": "SZ7uzWv1QKyxRGvI6tOv_Q"  }
{"type": "server", "timestamp": "2020-07-13T14:00:30,288Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "es-proxy", "node.name": "es-proxy-dedicated-master-node-01", "message": "Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.security-7][0], [.apm-agent-configuration][0], [.kibana_task_manager_1][0], [$kibana_yml_1][0]]]).", "cluster.uuid": "7PJtUzeYQ2SWOIgkb4Vr0Q", "node.id": "SZ7uzWv1QKyxRGvI6tOv_Q"  }

Is this configurations are correct ?

4

I don't see any indication why this cluster is not connected, sorry, but you should no longer be getting the message from your OP.

5

@DavidTurner Shall I turn log level to ALL and give output to you ?
Might be you can find something essential for me.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.