"None of the configured nodes are available" error when connecting to elastic cloud using x-pack

Elastic Stack Elasticsearch
1

I'm attempting to follow the elastic/found-shield-example (https://github.com/elastic/found-shield-example) to connect to a cluster running on elastic.co using the elastic Java client, however, I'm unable to connect successfully and get the following error when trying to run the example:

"Unable to get cluster health response: None of the configured nodes are available"

The cluster responds as follows when using curl:

curl -u user.name:password https://d44312.eu-west-1.aws.found.io:9243
{
  "name" : "instance-0000000005",
  "cluster_name" : "d44312",
  "cluster_uuid" : "nFyb_Hn6RPaA4NVsBsi8Pw",
  "version" : {
    "number" : "5.0.2",
    "build_hash" : "f6b4951",
    "build_date" : "2016-11-24T10:07:18.101Z",
    "build_snapshot" : false,
    "lucene_version" : "6.2.1"
  },
  "tagline" : "You Know, for Search"
}

The Java client is using version 5.0.2 as per the following gradle configuration:

repositories {
    maven {
        url "https://artifacts.elastic.co/maven"
    }
}

dependencies {

    compile "org.elasticsearch.client:transport:5.0.2"
    compile "org.elasticsearch.client:x-pack-transport:5.0.2"

    compile group: 'org.apache.logging.log4j', name: 'log4j-jcl', version: '2.7'
    compile group: 'org.apache.logging.log4j', name: 'log4j-api', version: '2.7'
    compile group: 'org.apache.logging.log4j', name: 'log4j-core', version: '2.7'
}

Following the elastic/found-shield-example code I use the following Settings object:

Settings settings = Settings.builder()
                .put("client.transport.nodes_sampler_interval", "5s")
                .put("client.transport.sniff", false)
                .put("transport.tcp.compress", true)
                .put("cluster.name", clusterName)
                .put("xpack.security.transport.ssl.enabled", enableSsl)
                .put("request.headers.X-Found-Cluster", "${cluster.name}")
                .put("xpack.security.user", System.getProperty("xpack.security.user"))
                .build();

EnableSsl defaults to true and ip6Enabled defaults to false. All other code is as per the example and I'm running the code with the following parameters:

-Dhost=d44312.eu-west-1.aws.found.io -Dxpack.security.user='user.name:password'

This results in clusterName == d44312 when passed to the settings object and System.getProperty("xpack.security.user") returning the correct user name and password as verified using curl. When I execute the application I get the following output:

2016-12-06 11:01:10 INFO  TransportExample:63 - Connecting to cluster: [d44312] via [d44312.eu-west-1.aws.found.io:9243] using ssl:[true]
2016-12-06 11:01:11 DEBUG InternalLoggerFactory:71 - Using SLF4J as the default logging framework
2016-12-06 11:01:13 DEBUG ThreadLocalRandom:71 - -Dio.netty.initialSeedUniquifier: 0x2fff87f07a81477e
2016-12-06 11:01:20 INFO  TransportExample:92 - Getting cluster health... 
2016-12-06 11:01:20 ERROR TransportExample:97 - Unable to get cluster health response: [None of the configured nodes are available: [{#transport#-1}{y8F-q5jKRxmNS2vq05ErBg}{176.34.254.82}{176.34.254.82:9243}, {#transport#-2}{XtAEiZuhTLqPaP6eyu_XdA}{46.137.179.48}{46.137.179.48:9243}, {#transport#-3}{q4Y3jgpVR8OMsUpJiSy5Sw}{46.137.180.129}{46.137.180.129:9243}, {#transport#-4}{MFvEEfS-R4qpJldiJhlX6g}{54.75.226.168}{54.75.226.168:9243}, {#transport#-5}{EBXGWf1sRkaWluepLYKjSQ}{54.217.237.247}{54.217.237.247:9243}, {#transport#-6}{FvNsCgSBRzqvZGvQtBUowQ}{54.217.243.67}{54.217.243.67:9243}]]

I'm somewhat stuck with this as I can't see where I'm going wrong. Could anyone suggest what the problem might be or recommend any avenues for investigation?

Thanks!

2

For the transport protocol I believe you should connect to port 9343 rather than 9243 as outlined here.

3

I did also try using 9343 with the found-shield-example but get similar results:

2016-12-06 11:43:40 INFO  TransportExample:63 - Connecting to cluster: [d44312] via [d44312.eu-west-1.aws.found.io:9343] using ssl:[true]
2016-12-06 11:43:46 INFO  TransportExample:92 - Getting cluster health... 
2016-12-06 11:43:46 ERROR TransportExample:97 - Unable to get cluster health response: [None of the configured nodes are available: [{#transport#-1}{nXl-WwaOTKm4LQ7kDUlhEw}{54.217.243.67}{54.217.243.67:9343}, {#transport#-2}{qHD6KfhiRz-EFIxzUrowPg}{176.34.254.82}{176.34.254.82:9343}, {#transport#-3}{ASz16CviRA-nbjGzM7sJIA}{46.137.179.48}{46.137.179.48:9343}, {#transport#-4}{zBJWR40yR7SEtFhBCny0og}{46.137.180.129}{46.137.180.129:9343}, {#transport#-5}{BFxT11SiS1G71Yuzrny_VQ}{54.75.226.168}{54.75.226.168:9343}, {#transport#-6}{DO5buQCQSbmHcYZ2TVZXlw}{54.217.237.247}{54.217.237.247:9343}]]{54.217.237.247:9343}]]

I'm building the TransportClient using the PreBuiltXPackTransportClient like this:

TransportClient client = new PreBuiltXPackTransportClient(settings);

Your linked example seems to uses a different builder to construct the TransportClient but on inspection the PreBuiltXpackTransportClient just seems to do what your example does under the hood:

public PreBuiltXPackTransportClient(Settings settings, Collection<Class<? extends Plugin>> plugins, HostFailureListener hostFailureListener) {
        super(settings, addPlugins(plugins, Collections.singletonList(XPackPlugin.class)), hostFailureListener);
    }
4

I'm not sure if this is relevant but the _nodes response from my cluster has a completely different IP address to anything listed in the list of configured nodes returned by the logging.

curl -u user.name:password https://d44312.eu-west-1.aws.found.io:9243/_nodes
...
"transport": {
    "bound_address": [
        "172.17.0.62:19426"
    ],
    "profiles": {},
    "publish_address": "10.81.6.227:19426"
}
5

Hi @Dan_Bardsley,

I've edited your above posts to only show the first 6 characters of you cluster ID for security purposes.

Using the https://github.com/elastic/found-shield-example example, I have added an internal user to your cluster and tested with that user and it works as expected with the following:

mvn clean compile exec:java -Dhost=d44312.eu-west-1.aws.found.io -Dxpack.security.user='<username>:<password>

I then used your example username and password and I get the following error unable to authenticate user [user.name].

Can you run the example with a username and password that you know the details (I'm assuming you might not have provided the username/password details for security).

6

Interesting. I removed the credentials from my posts for security reasons - the accounts I'm actually running with are correct. Rather than running the example with my gradle configuration I tried using the maven command directly against the found-shield-example repo. I still didn't manage to connect but I'm now getting the following exception:

Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (48,54,54,50)
	at org.elasticsearch.transport.TcpTransport.validateMessageHeader(TcpTransport.java:1103) ~[elasticsearch-5.0.2.jar:5.0.2]
	at org.elasticsearch.transport.netty4.Netty4SizeHeaderFrameDecoder.decode(Netty4SizeHeaderFrameDecoder.java:36) ~[transport-netty4-client-5.0.2.jar:5.0.2]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411) ~[netty-codec-4.1.5.Final.jar:4.1.5.Final]
	... 22 more

I'm running this on OSX 10.10.5 if that makes any difference with this Java version:

java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
7

Suffering the same problem here.

8

@Dan_Bardsley - ok, let me look into this with the team to see if we can replicate the issue.

@soundlly can you provide the first 6 characters of your cluster ID to investigate the issue.

Christian

9

@cstrzadala This is my cluster ID : 453c29
Thank you for support.

10

I'm still looking into this but I'm now able to connect to my cluster from the Java client. It appears to be an issue with the dependancies that maven was pulling down but I've yet to confirm this completely.

Running:

mvn dependency:purge-local-repository

Then:

mvn clean compile exec:java -Dhost=xxxxxxxxxxxxxxx.eu-west-1.aws.found.io -Dxpack.security.user=xxxxxxx:xxxxx

got things working for me but I don't understand why I was seeing issues yesterday.

11

@Dan_Bardsley ok, that's interesting. Seems that without a mvn clean then the previously compiled code with perhaps an older 5.0.0 Elasticsearch dependency was being used.

So to confirm, your able to connect your cluster with the transport client?

12

Hi @soundlly,

Can you try the https://github.com/elastic/found-shield-example example with the following command:

mvn clean compile exec:java -Dhost=xxxxxxxxxxxxxxx.us-west-2.aws.found.io -Dxpack.security.user=xxxxxxx:xxxxx

Would like to see if a clean will help your issue as well.

Christian

13

@cstrzadala - That's correct. I can connect to the cluster with the 5.0.2 transport client. I'm still trying to work out exactly why I couldn't yesterday!

14

@cstrzadala - I tried as Dan, but I failed to connect to server.
I cleaned git project but it was in vain.
Do you have another suggestion?

I've done my job with using REST API, but I'd like to finish debugging the issue.

Thank you.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.