[ERROR][discovery.ec2 ] [host] unexpected error while joining cluster, trying again org.elasticsearch.ElasticsearchException: Ping execution failed

Elastic Stack Elasticsearch
1

Hello all

Today we had a very strange error appear today in our cluster that is preventing the node from connecting. Below is our errors


[2024-02-06 23:18:11,590][ERROR][discovery.ec2            ] [Windeagle] unexpected error while joining cluster, trying again
org.elasticsearch.ElasticsearchException: Ping execution failed
        at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.ping(UnicastZenPing.java:260)
        at org.elasticsearch.discovery.zen.ping.ZenPingService.ping(ZenPingService.java:146)
        at org.elasticsearch.discovery.zen.ping.ZenPingService.pingAndWait(ZenPingService.java:124)
        at org.elasticsearch.discovery.zen.ZenDiscovery.findMaster(ZenDiscovery.java:1007)
        at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:361)
        at org.elasticsearch.discovery.zen.ZenDiscovery.access$6100(ZenDiscovery.java:86)
        at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1384)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.SecurityException: Invalid SSL master secret
        at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.verifyMasterSecret(SdkTLSSocketFactory.java:193)
        at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:128)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
        at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:728)
        at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
        at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:11783)
        at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5905)
        at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider.buildDynamicNodes(AwsEc2UnicastHostsProvider.java:110)
        at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.sendPings(UnicastZenPing.java:313)
        at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.ping(UnicastZenPing.java:219)
        ... 9 more

We have made no changes to the servers and where working earlier today. However, this errors keeps occurring even after yaml changes, service restarts, and server restarts. I even built a new EC2 instance and it is receiving the same error.

The odd thing is we have never used SSL and even added the SSL requirement to false in the yaml.

Any help would be greatly appreciated.

2

What version of Elasticsearch is this? It appears to be very old and running a very old version of the AWK SDK.

3

We are on a very old version of elasticsearch. We are running Elasticsearch 1.7.6.

This is the AWS version installed -

aws-cli/1.18.147 Python/2.7.18 Linux/5.10.198-187.748.amzn2.x86_64 botocore/1.18.6```
4

That's so old that I doubt you're going to find anyone to help.

From the error, it looks like the Java AWS SDK that ES is using for EC2 discovery has some sort of issue handshaking with some URL - perhaps the EC2 metadata service.

It doesn't appear to be an Elasticsearch specific issue

5

Yeah. From the top of my head, I remember that upgrading the AWS SDK was mandatory at some point because it was not compatible anymore with their APIS.

The only chance is to upgrade.
Ideally to 8.12 but I know this is a huge step as so many things changed, like the mapping, the way the ec2 plugin works and is configured....

6

Thanks. That is what I figured but was hoping it wasn't. As a bandaid for now, what version do I upgrade the sdk?

7

I can't tell but the last update I can see is from 2 years ago with 1.12.270.

Not sure it will be compatible with the 1.x version though.

8

Okay thank you for all this help. I'll try some things

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.