I want to set up a cluster on ECS and am having some trouble with EC2 discovery. The plugin is timing out while trying to retrieve a list of available instances. Running the same command in aws-cli returns very quickly and I'm not sure what the difference is between their configs. I can curl the nodes in the other containers manually as well.
Any help would be appreciated.
I've turned trace logging on for the plugin and these are the results:
18:11:23.952 [elasticsearch[uhFXbz7][generic][T#1]] DEBUG com.amazonaws.auth.AWSCredentialsProviderChain - Loading credentials from InstanceProfileCredentialsProvider
18:11:23.965 [elasticsearch[uhFXbz7][generic][T#1]] DEBUG com.amazonaws.request - Sending Request: POST https://ec2.us-east-1.amazonaws.com / Parameters: ({"Action":["DescribeInstances"],"Version":["2015-10-01"],"Filter.1.Name":["instance-state-name"],"Filter.1.Value.1":["runn ing"],"Filter.1.Value.2":["pending"],"Filter.2.Name":["tag:instance-type"],"Filter.2.Value.1":["cdi-logging"]}Headers: (User-Agent: aws-sdk-java/1.10.69 Linux/3.10.0-514.10.2.el7.x86_64 OpenJDK_64-Bit_Server_VM/25.92-b14/1.8.0_92-internal, amz-sdk-invocation-id: 6f071a69-472e-4237-8e96-8681f29f71cc, )
18:11:23.971 [elasticsearch[uhFXbz7][generic][T#1]] DEBUG com.amazonaws.auth.AWS4Signer - AWS4 Canonical Request: '"POST /
amz-sdk-invocation-id:6f071a69-472e-4237-8e96-8681f29f71cc
amz-sdk-retry:0/0/
host:ec2.us-east-1.amazonaws.com
user-agent:aws-sdk-java/1.10.69 Linux/3.10.0-514.10.2.el7.x86_64 OpenJDK_64-Bit_Server_VM/25.92-b14/1.8.0_92-internal
x-amz-date:20170322T181123Z
x-amz-security-token:<REDACTED>
amz-sdk-invocation-id;amz-sdk-retry;host;user-agent;x-amz-date;x-amz-security-token"
18:11:23.971 [elasticsearch[uhFXbz7][generic][T#1]] DEBUG com.amazonaws.auth.AWS4Signer - AWS4 String to
Sign: <REDACTED>
18:11:24.051 [elasticsearch[uhFXbz7][generic][T#1]] DEBUG com.amazonaws.http.conn.ssl.SdkTLSSocketFactory - socket.getSupportedProtocols(): [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2], socket.getEnabledProtocols(): [TLSv1, TLSv1.1, TLSv1.2]
18:11:24.051 [elasticsearch[uhFXbz7][generic][T#1]] DEBUG com.amazonaws.http.conn.ssl.SdkTLSSocketFactory - TLS protocol enabled for SSL handshake: [TLSv1.2, TLSv1.1, TLSv1]
18:11:24.051 [elasticsearch[uhFXbz7][generic][T#1]] DEBUG com.amazonaws.http.conn.ssl.SdkTLSSocketFactory - connecting to ec2.us-east-1.amazonaws.com/54.239.29.8:443
18:12:14.100 [elasticsearch[uhFXbz7][generic][T#1]] INFO com.amazonaws.http.AmazonHttpClient - Unable to
execute HTTP request: connect timed out
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_92-internal]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_92-internal]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_92-internal]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_92-internal]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_92-internal]
at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_92-internal]
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) ~[?:?]
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:542) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:412) ~[httpclient-4.5.2.jar:4.5.2]
at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:134) ~[aws-java-sdk-core-1.10.69.jar:?]
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:179) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:328) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:612) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:447) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:884) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) ~[httpclient-4.5.2.jar:4.5.2]
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:837) ~[aws-java-sdk-core-1.10.69.jar:?]
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:607) [aws-java-sdk-core-1.10.69.jar:?]
at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:376) [aws-java-sdk-core-1.10.69.jar:?]
at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:338) [aws-java-sdk-core-1.10.69.jar:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:287) [aws-java-sdk-core-1.10.69.jar:?]
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:11133) [aws-java-sdk-ec2-1.10.69.jar:?]
at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5403) [aws-java-sdk-ec2-1.10.69.jar:?]
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider.fetchDynamicNodes(AwsEc2UnicastHostsProvider.java:120) [discovery-ec2-5.2.2.jar:5.2.2]
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider$DiscoNodesCache.refresh(AwsEc2UnicastHostsProvider.java:238) [discovery-ec2-5.2.2.jar:5.2.2]
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider$DiscoNodesCache.refresh(AwsEc2UnicastHostsProvider.java:223) [discovery-ec2-5.2.2.jar:5.2.2]
at org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54) [elasticsearch-5.2.2.jar:5.2.2]
My config is as follows:
network.host: 0.0.0.0
network.publish_host: _ec2:privateIp_
bootstrap.memory_lock: true
xpack.security.enabled: false
plugin.mandatory: discovery-ec2
discovery.zen.hosts_provider: ec2
discovery.zen.minimum_master_nodes: 2
discovery.ec2.host_type: private_ip
discovery.ec2.tag.instance-type: <REDACTED>
cloud.aws.read_timeout: 3000s
cloud.aws.region: us-east-1
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone