Unable to connect to Remote Elastic Setup via Transport Client

I am working on ElasticSearch version 6.6.0, Single Node Cluster running on a remote VM. I wanted to connect Java Transport Client to the remote ES using the tcp port 9300. I was successful in opening the tcp port to be accessible from my machine (tested using telnet to port 9200 and 9300). Also able to get the Elastic Details on http://<public_ip>:9200/.

I am at my wits end not able to solve this mystery of why the transport client is not able to connect to the Cluster for indexing/querying.

elasticsearch.yml looks like this:

cluster.name: test-elastic-surabhi
node.name: ${HOSTNAME}

network.host: ${HOSTNAME}

action.auto_create_index: .monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*

http://public_ip:9200 results in:

{
  "name" : "test-elastic-surabhi-1",
  "cluster_name" : "test-elastic-surabhi",
  "cluster_uuid" : "c-w5p531Szqsv_up3OS_Uw",
  "version" : {
    "number" : "6.6.0",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "a9861f4",
    "build_date" : "2019-01-24T11:27:09.439740Z",
    "build_snapshot" : false,
    "lucene_version" : "7.6.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

telnet to 9300 and 9200 from my local machine also works.

pom.xml points to correct version of ES:

     <dependency>
	<groupId>org.elasticsearch</groupId>
			<artifactId>elasticsearch</artifactId>
			<version>6.6.0</version>
		</dependency>

		<!-- https://mvnrepository.com/artifact/org.elasticsearch.client/transport -->
		<dependency>
			<groupId>org.elasticsearch.client</groupId>
			<artifactId>transport</artifactId>
			<version>6.6.0</version>
		</dependency>

Here's the code to connect to TCP from Transport Client:

Settings settings = Settings.builder().put("cluster.name", DataStoreSettings.ES_CLUSTER_NAME)
        .put("client.transport.sniff", true).build();
esTransportClient = new PreBuiltTransportClient(settings);
for (String eachHostPort : DataStoreSettings.ES_HOST) {
        LOGGER.info("adding [{}] to TransportClient ... ", eachHostPort);
        String[] hostPortTokens = eachHostPort.split(":");
        if (hostPortTokens.length < 2)
          throw new Exception(
              "ERROR: bad ElasticSearch host:port configuration - wrong format: " + eachHostPort);
        int port = 9300; // default ES port
        try {
          port = Integer.parseInt(hostPortTokens[1].trim());
        } catch (Exception e) {
          LOGGER.error("ERROR parsing port from the ES config [{}]- using default port 9300",
              eachHostPort);
        }
        esTransportClient.addTransportAddress(
            new TransportAddress(InetAddress.getByName(hostPortTokens[0].trim()), port));
      }

But from the Transport Client used in my Java Code i can see exceptions

2019-02-14 22:31:23,599 TRACE [TransportLogger]: an exception occurred formatting a WRITE trace message
java.io.EOFException: tried to read: 105 bytes but only 27 remaining
        at org.elasticsearch.common.bytes.BytesReferenceStreamInput.ensureCanReadBytes(BytesReferenceStreamInput.java:121) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper.ensureCanReadBytes(BytesReference.java:283) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.common.io.stream.StreamInput.readArraySize(StreamInput.java:1026) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.common.io.stream.StreamInput.readString(StreamInput.java:374) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.common.io.stream.StreamInput.readStringArray(StreamInput.java:469) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.transport.TransportLogger.format(TransportLogger.java:101) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.transport.TransportLogger.logOutboundMessage(TransportLogger.java:55) ~[elasticsearch-6.6.0.jar:6.6.0]

Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: <private_ip>/<private_ip>:9300

2019-02-14 22:31:54,590 DEBUG [TransportClientNodesService]: failed to connect to discovered node [{test-elastic-surabhi-1}{F_we0jg4Q7uoQ3uOaDvv-w}{kfqF3aRcQ7mWAxhXv9R_uQ}{<private_ip>}{<private_ip>:9300}{ml.machine_memory=7673036800, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]
org.elasticsearch.transport.ConnectTransportException: [test-elastic-surabhi-1][<private_ip>:9300] connect_timeout[30s]
        at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onTimeout(TcpTransport.java:1576) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-6.6.0.jar:6.6.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_191]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_191]
        at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_191]
2019-02-14 22:57:38,609 ERROR [Processor]: postBulkToEs: NoNodeAvailableException. ex: {}
org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{Cn4_xs_OQdeo3-2l5UgAGQ}{test-elastic-surabhi-1}{<public_ip>:9300}]
        at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:351) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:249) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:60) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:388) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:391) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:46) ~[elasticsearch-6.6.0.jar:6.6.0]
        at com.shieldsquare.nozzle.Processor.postBulkToEs(Processor.java:152) [classes/:?]

Also logs of the Remote Elastic Single Node Cluster prints:

publish_address {<private_ip>:9300}, bound_addresses {<private_ip>:9300}

You need to add the cluster name in your client settings.

Note that you should move away from TransportClient and use Rest Client instead.

I am adding the cluster name to the Transport Client Setting.

Here is the Client Connection Code:

Settings settings = Settings.builder().put("cluster.name", DataStoreSettings.ES_CLUSTER_NAME)
        .put("client.transport.sniff", true).build();
esTransportClient = new PreBuiltTransportClient(settings);
for (String eachHostPort : DataStoreSettings.ES_HOST) {
        LOGGER.info("adding [{}] to TransportClient ... ", eachHostPort);
        String[] hostPortTokens = eachHostPort.split(":");
        if (hostPortTokens.length < 2)
          throw new Exception(
              "ERROR: bad ElasticSearch host:port configuration - wrong format: " + eachHostPort);
        int port = 9300; // default ES port
        try {
          port = Integer.parseInt(hostPortTokens[1].trim());
        } catch (Exception e) {
          LOGGER.error("ERROR parsing port from the ES config [{}]- using default port 9300",
              eachHostPort);
        }
        esTransportClient.addTransportAddress(
            new TransportAddress(InetAddress.getByName(hostPortTokens[0].trim()), port));
      }

Ha sorry. Missed it.

Note that the host you are using must be the <private_ip>. Is it?

BTW I hope you did not open your cluster on internet...

Using the hostname of the machine, GCP internally resolves it to the private ip of the machine.

Yes I am trying to connect it over the internet via the public ip and port 9300. i can see in the exception logs of the transport client that it is converting to internal ip (which i assume is the publish_host) and hence not able to connect. Is there a way around it?

Do you have any security layer?

Is there a way around it?

Not sure. If you have a proxy and that proxy redirects 9300 public traffic to the 9300 private one, that might work but I'm unsure.

Again, I'd not use the TransportClient but the RestClient.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.