Hello All,
I'm setting up a new Elasticsearch cluster with 3 nodes out of which 3 are master nodes and data nodes as well.
The following is the config of the ES
# ======================== Elasticsearch Configuration =========================
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: test_cluster
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: ip-10-0-0-21
node.master: true
node.data: true
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
bootstrap.system_call_filter: false
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 10.0.0.21
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.zen.ping.unicast.hosts: ["10.0.0.21", "10.0.0.22", "10.0.0.24"]
cluster.initial_master_nodes: ["10.0.0.21", "10.0.0.22", "10.0.0.24"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 2
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
The issue here is that even though i had 3 nodes mentioned in discovery.zen.ping.unicast.hosts: ["10.0.0.21", "10.0.0.22", "10.0.0.24"]
But the cluster is forming with only one node the following is the logs
[2020-11-18T21:16:33,017][INFO ][o.e.e.NodeEnvironment ] [ip-10-0-0-21] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [185.7gb], net total_space [193.6gb], types [ext4]
[2020-11-18T21:16:33,021][INFO ][o.e.e.NodeEnvironment ] [ip-10-0-0-21] heap size [990.7mb], compressed ordinary object pointers [true]
[2020-11-18T21:16:33,182][INFO ][o.e.n.Node ] [ip-10-0-0-21] node name [ip-10-0-0-21], node ID [KZvV5oyHQaCpAVxbuc-Prw], cluster name [Cognizer_dev]
[2020-11-18T21:16:33,183][INFO ][o.e.n.Node ] [ip-10-0-0-21] version[7.6.2], pid[25581], build[default/deb/ef48eb35cf30adf4db14086e8aabd07ef6fb113f/2020-03-26T06:34:37.794943Z], OS[Linux/5.4.0-1029-gcp/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13.0.2/13.0.2+8]
[2020-11-18T21:16:33,184][INFO ][o.e.n.Node ] [ip-10-0-0-21] JVM home [/usr/share/elasticsearch/jdk]
[2020-11-18T21:16:33,184][INFO ][o.e.n.Node ] [ip-10-0-0-21] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-7399941801453188432, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2020-11-18T21:16:35,192][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [aggs-matrix-stats]
[2020-11-18T21:16:35,192][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [analysis-common]
[2020-11-18T21:16:35,193][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [flattened]
[2020-11-18T21:16:35,193][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [frozen-indices]
[2020-11-18T21:16:35,193][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [ingest-common]
[2020-11-18T21:16:35,193][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [ingest-geoip]
[2020-11-18T21:16:35,194][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [ingest-user-agent]
[2020-11-18T21:16:35,194][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [lang-expression]
[2020-11-18T21:16:35,194][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [lang-mustache]
[2020-11-18T21:16:35,195][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [lang-painless]
[2020-11-18T21:16:35,195][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [mapper-extras]
[2020-11-18T21:16:35,195][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [parent-join]
[2020-11-18T21:16:35,195][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [percolator]
[2020-11-18T21:16:35,196][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [rank-eval]
[2020-11-18T21:16:35,196][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [reindex]
[2020-11-18T21:16:35,196][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [repository-url]
[2020-11-18T21:16:35,196][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [search-business-rules]
[2020-11-18T21:16:35,197][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [spatial]
[2020-11-18T21:16:35,197][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [systemd]
[2020-11-18T21:16:35,197][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [transform]
[2020-11-18T21:16:35,198][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [transport-netty4]
[2020-11-18T21:16:35,198][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [vectors]
[2020-11-18T21:16:35,198][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [x-pack-analytics]
[2020-11-18T21:16:35,198][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [x-pack-ccr]
[2020-11-18T21:16:35,199][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [x-pack-core]
[2020-11-18T21:16:35,199][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [x-pack-deprecation]
[2020-11-18T21:16:35,199][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [x-pack-enrich]
[2020-11-18T21:16:35,200][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [x-pack-graph]
[2020-11-18T21:16:35,200][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [x-pack-ilm]
[2020-11-18T21:16:35,200][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [x-pack-logstash]
[2020-11-18T21:16:35,200][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [x-pack-ml]
[2020-11-18T21:16:35,201][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [x-pack-monitoring]
[2020-11-18T21:16:35,201][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [x-pack-rollup]
[2020-11-18T21:16:35,201][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [x-pack-security]
[2020-11-18T21:16:35,201][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [x-pack-sql]
[2020-11-18T21:16:35,202][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [x-pack-voting-only-node]
[2020-11-18T21:16:35,202][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] loaded module [x-pack-watcher]
[2020-11-18T21:16:35,203][INFO ][o.e.p.PluginsService ] [ip-10-0-0-21] no plugins loaded
[2020-11-18T21:16:38,995][INFO ][o.e.x.s.a.s.FileRolesStore] [ip-10-0-0-21] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2020-11-18T21:16:39,870][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [ip-10-0-0-21] [controller/25714] [Main.cc@110] controller (64 bit): Version 7.6.2 (Build e06ef9d86d5332) Copyright (c) 2020 Elasticsearch BV
[2020-11-18T21:16:40,564][DEBUG][o.e.a.ActionModule ] [ip-10-0-0-21] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2020-11-18T21:16:40,699][INFO ][o.e.d.DiscoveryModule ] [ip-10-0-0-21] using discovery type [zen] and seed hosts providers [settings]
[2020-11-18T21:16:41,810][INFO ][o.e.n.Node ] [ip-10-0-0-21] initialized
[2020-11-18T21:16:41,811][INFO ][o.e.n.Node ] [ip-10-0-0-21] starting ...
[2020-11-18T21:16:41,987][INFO ][o.e.t.TransportService ] [ip-10-0-0-21] publish_address {10.0.0.21:9300}, bound_addresses {10.0.0.21:9300}
[2020-11-18T21:16:42,371][INFO ][o.e.b.BootstrapChecks ] [ip-10-0-0-21] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-11-18T21:16:42,410][INFO ][o.e.c.c.Coordinator ] [ip-10-0-0-21] cluster UUID [cO2QuJrMR1e9ze3BC1XLbg]
[2020-11-18T21:16:42,576][INFO ][o.e.c.s.MasterService ] [ip-10-0-0-21] elected-as-master ([1] nodes joined)[{ip-10-0-0-21}{KZvV5oyHQaCpAVxbuc-Prw}{GRUinyG9QvyYBc-ZGFQOog}{10.0.0.21}{10.0.0.21:9300}{dilm}{ml.machine_memory=16790884352, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 7, version: 45, delta: master node changed {previous [], current [{ip-10-0-0-21}{KZvV5oyHQaCpAVxbuc-Prw}{GRUinyG9QvyYBc-ZGFQOog}{10.0.0.21}{10.0.0.21:9300}{dilm}{ml.machine_memory=16790884352, xpack.installed=true, ml.max_open_jobs=20}]}
[2020-11-18T21:16:42,671][INFO ][o.e.c.s.ClusterApplierService] [ip-10-0-0-21] master node changed {previous [], current [{ip-10-0-0-21}{KZvV5oyHQaCpAVxbuc-Prw}{GRUinyG9QvyYBc-ZGFQOog}{10.0.0.21}{10.0.0.21:9300}{dilm}{ml.machine_memory=16790884352, xpack.installed=true, ml.max_open_jobs=20}]}, term: 7, version: 45, reason: Publication{term=7, version=45}
[2020-11-18T21:16:42,742][INFO ][o.e.h.AbstractHttpServerTransport] [ip-10-0-0-21] publish_address {10.0.0.21:9200}, bound_addresses {10.0.0.21:9200}
[2020-11-18T21:16:42,743][INFO ][o.e.n.Node ] [ip-10-0-0-21] started
[2020-11-18T21:16:42,971][INFO ][o.e.l.LicenseService ] [ip-10-0-0-21] license [5f2de71e-b085-40de-8512-b62daa4f6810] mode [basic] - valid
[2020-11-18T21:16:42,973][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [ip-10-0-0-21] Active license is now [BASIC]; Security is disabled`Preformatted text`
[2020-11-18T21:16:42,988][INFO ][o.e.g.GatewayService ] [ip-10-0-0-21] recovered [1] indices into cluster_state
[2020-11-18T21:16:43,589][INFO ][o.e.c.r.a.AllocationService] [ip-10-0-0-21] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.security-7][0]]]).
can anyone tell me what am i doing wrong.