Elasticsearch ec2 plugin configuration fail

Elastic Stack Elasticsearch
1

I am trying to configure an elasticsearch cluster in aws using the ec2 discovery plugin. I am running on Elasticsearch 7.6 . I have attached an instance-profile that has all the access for ec2. The logs are saying ** master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{ec2:privateDns}**

My elasticsearch.yml is:
cluster.name: "elasticsearch-ndb"
node.name: ec2:privateDns
network.host: ec2
discovery.zen.hosts_provider: ec2
discovery.ec2.tag.Elastic: elasticsearch-ndb
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery.ec2.availability_zones: us-east-1
discovery.ec2.host_type: private_ip
discovery.ec2.any_group: true
xpack.security.enabled: false
discovery.zen.minimum_master_nodes: 2

2

This setting is required on at least one (ideally all) master-eligible nodes the first time the cluster forms.

3

How would this work in the instance of AWS. I wont know the IP or be really able to set the master nodes without know the IP and that is not generated until the Instances are up.

4

The IP address is not relevant. This setting should contain the names of the first set of master nodes, which you should fix before starting the cluster. Unfortunately there's no way to auto-discover this using the AWS APIs so it must be set explicitly. Once the cluster is running you should remove this setting.

5

Is there a curl command I can use to set it, it seems weird that I cant do it in the yml. If i used master.node: true would that work as well?

6

also this article says the plugin it used to auto discover master nodes? https://www.elastic.co/guide/en/elasticsearch/plugins/current/discovery-ec2-usage.html

7

You are perhaps confusing discovery with cluster bootstrapping? They are quite different concepts. The discovery-ec2 plugin does indeed allow you to auto-discover the master-eligible nodes in the cluster but does not have any influence over cluster bootstrapping.

8

Do you by chance have an example Elasticsearch.yml that I can follow that would allow me to use the ec2 plugin also get the masters to join.

9

Something like this perhaps?

# needed for cluster bootstrapping, only until the cluster has formed
# and only on the first set of master-eligible nodes
node.name: master-0
cluster.initial_master_nodes:
  - master-0
  - master-1
  - master-2

# needed for discovery, on all nodes
discovery.seed_providers: ec2
discovery.ec2.availability_zones:
  - us-east-1a
  - us-east-1b
  - us-east-1c
discovery.ec2.tag.elasticsearch_role: master

And similarly for the other two masters, just changing the node.name.

On data nodes you just need the last three settings, and that's also true for master-eligible nodes once the cluster has formed for the first time.

2 Likes
10

Hi David,

Before opening another thread I want to understand if its the same issue:

[2020-02-23T10:08:20,864][WARN ][o.e.c.c.ClusterFormationFailureHelper] [ip-172-31-29-135] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{ip-172-31-29-135}{1PueDBkSR7eQm9uTQrJqvA}{ivBbrdlqRFa2Y7-2ANeZ1A}{172.31.29.135}{172.31.29.135:9300}{lm}{aws_availability_zone=us-east-2b, ml.machine_memory=3885060096, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, [::1]:9305, 172.31.26.186:9300, 172.31.29.246:9300, 172.31.29.187:9300, 172.31.29.135:9300] from hosts providers and [{ip-172-31-29-135}{1PueDBkSR7eQm9uTQrJqvA}{ivBbrdlqRFa2Y7-2ANeZ1A}{172.31.29.135}{172.31.29.135:9300}{lm}{aws_availability_zone=us-east-2b, ml.machine_memory=3885060096, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0

with config:

bootstrap.memory_lock: true
node.name: ${HOSTNAME}

action.destructive_requires_name: true
indices.fielddata.cache.size: 1% # default is unbounded
cluster.name: ZZZZZZ

# only data nodes should have ingest and http capabilities
node.master: true
node.data: false
node.ingest: false
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.authc.api_key.enabled: true
xpack.monitoring.enabled: true
##
xpack.security.http.ssl.keystore.path: elastic-certificates.p12
xpack.security.http.ssl.truststore.path: elastic-certificates.p12
xpack.security.http.ssl.client_authentication: optional
##
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
##
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

network.host: _ec2:privateIpv4_,localhost
plugin.mandatory: discovery-ec2
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery:
    seed_providers: ec2
    ec2.groups: ZZZZZZ
    ec2.host_type: private_ip
    ec2.tag.Cluster: ZZZZZZZZZ
    ec2.availability_zones: us-east-2a,us-east-2b,us-east-2c
    ec2.protocol: http # no need in HTTPS for internal AWS calls
    ec2.endpoint: ec2.us-east-2.amazonaws.com

Version:

curl -k https://localhost:9200  -H "Authorization: BasicZZZZZZZZZZZ"
{
  "name" : "ip-172-31-29-135",
  "cluster_name" : "ZZZZZZZ",
  "cluster_uuid" : "oXxZ_JERSR-9fhqwSu69Qw",
  "version" : {
    "number" : "7.6.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "7f634e9f44834fbc12724506cc1da681b0c3b1e3",
    "build_date" : "2020-02-06T00:09:00.449973Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Error from cluster:

curl -k https://localhost:9200/_cat/nodes  -H "Authorization: BasicZZZZZZ"
{"error":
{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],
"type":"master_not_discovered_exception",
"reason":null},
"status":503}

Thanks.

11

Looks related, @scriptdb, although it's normally best to open a new post anyway. Your log also says:

As I said above, cluster.initial_master_nodes is required the first time the cluster forms.

12

Thanks David. cluster.initial_master_nodes is only for the initial start? if master nodes crash and rebuilt will this configuration need to be inserted dynamically to elasticsearch.yml on all running nodes?

13

Yes, only for the initial start.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.