Discovery EC2 Failed When Adding More Nodes ES 5.2.1

[ritchierich (Rich Horace)]

I got the following Errors while trying to add more Elasticsearch nodes to my v5.2.1 Cluster with EC2 Discovery.

My solution was to replace discovery.ec2 with set the IPs with discovery.zen.ping.unicast.hosts.

Below are the errors and my elastic search.yml. Let me know if you need any other details. Thanks in advance.

ERRORS:

2017-02-24T17:53:00,799][INFO ][o.e.n.Node               ] [prod_elastic_cluster_5_2-04] starting ...
[2017-02-24T17:53:00,968][WARN ][i.n.u.i.MacAddressUtil   ] Failed to find a usable hardware address from the network interfaces; using random bytes: 2d:00:a6:1e:f7:f5:9e:1c
[2017-02-24T17:53:01,151][INFO ][o.e.t.TransportService   ] [prod_elastic_cluster_5_2-04] publish_address {10.XXX.XXX.XXX:9300}, bound_addresses {10.XXX.XXX.XXX:9300}
[2017-02-24T17:53:01,156][INFO ][o.e.b.BootstrapChecks    ] [prod_elastic_cluster_5_2-04] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-02-24T17:53:10,853][INFO ][o.e.x.m.e.Exporters      ] [prod_elastic_cluster_5_2-04] skipping exporter [default_local] as it is not ready yet
[2017-02-24T17:53:20,856][INFO ][o.e.x.m.e.Exporters      ] [prod_elastic_cluster_5_2-04] skipping exporter [default_local] as it is not ready yet
[2017-02-24T17:53:30,859][INFO ][o.e.x.m.e.Exporters      ] [prod_elastic_cluster_5_2-04] skipping exporter [default_local] as it is not ready yet
[2017-02-24T17:53:31,211][WARN ][o.e.n.Node               ] [prod_elastic_cluster_5_2-04] timed out while waiting for initial discovery state - timeout: 30s
[2017-02-24T17:53:31,218][INFO ][o.e.h.HttpServer         ] [prod_elastic_cluster_5_2-04] publish_address {10.XXX.XXX.XXX:9200}, bound_addresses {10.XXX.XXX.XXX:9200}
[2017-02-24T17:53:31,219][INFO ][o.e.n.Node               ] [prod_elastic_cluster_5_2-04] started
[2017-02-24T17:53:40,862][INFO ][o.e.x.m.e.Exporters      ] [prod_elastic_cluster_5_2-04] skipping exporter [default_local] as it is not ready yet
[2017-02-24T17:53:50,864][INFO ][o.e.x.m.e.Exporters      ] [prod_elastic_cluster_5_2-04] skipping exporter [default_local] as it is not ready yet
[2017-02-24T17:54:00,867][INFO ][o.e.x.m.e.Exporters      ] [prod_elastic_cluster_5_2-04] skipping exporter [default_local] as it is not ready yet

Elasticsearch.yml Config

cluster.name: "My_Cluster"
node.name: "My_Cluster_Node"
node.master: "false"
node.data: "true"
node.ingest: "true"
bootstrap.memory_lock: true
network.host: "10.xxx.xxx.xxx"
path:
  data:
    - "/mnt/data/data1"
    - "/mnt/data/data2" 

discovery.zen.minimum_master_nodes: "2"

# Memory Limits
indices.fielddata.cache.size: 40%
indices.breaker.fielddata.limit: 60% 
indices.breaker.request.limit: 40%
indices.breaker.total.limit: 70%

#script.disable_dynamic: true
http.cors.enabled: true
http.cors.allow-origin: "*"

######## EC2 Discovery ##################

cloud.aws.region: us-west-2
cloud.aws.protocol: https
cloud.aws.ec2.protocol: https
cloud.aws.read_timeout: 30s

discovery.zen.hosts_provider: ec2
discovery.ec2.any_group: "false"
discovery.ec2.groups: "My_Cluster_Sec_Group"
discovery.ec2.host_type: "private_ip"
########################################

Solution:

discovery.zen.ping.unicast.hosts:
   - 10.XXX.XXX.XXX
   - 10.XXX.XXX.XXX
   - 10.XXX.XXX.XXX
   - 10.XXX.XXX.XXX
   - 10.XXX.XXX.XXX

[dadoonet (David Pilato)]

Please format your code using </> icon as explained in this guide. It will make your post more readable.

Or use markdown style like:

```
CODE
```

I modified your post.

May be you can get more traces by changing the log level for org.elasticsearch.discovery.ec2 and org.elasticsearch.cloud.aws to TRACE?

[ritchierich (Rich Horace)]

Thanks David. Formatting looks great.

This is the first time I've run into any issues with EC2 Discovery. Also, I was trying to add nodes to an existing cluster already configured with EC2 Discovery.

I'll try testing with more logging. Keep you posted.

Thanks,
Rich