Elasticsearch randomly switched initial master node

Elastic Stack Elasticsearch
1

I'm rather inexperienced with the deeper workings of Elasticsearch, but I've been able to establish a two-node cluster with a small (~20gb) index of text files. I'm running Elasticsearch 7.6.2 on Windows Server 2012. Prior to this morning, I would start node-1 and then node-2, as I designated node-1 as the initial master node. Once both nodes fully loaded, the cluster health would turn green and it would function.

For some reason, it seems Elastic is now designating node-2 as the initial master. When I start node-1 first, the following logs are generated, which I believe mean that Elastic is waiting for node-2 to start:

[2020-05-28T13:18:23,237][INFO ][o.e.e.NodeEnvironment    ] [node-1] using [1] data paths, mounts [[DATA-E (Z:)]], net usable_space [62.5gb], net total_space [35.9tb], types [NTFS]
[2020-05-28T13:18:23,252][INFO ][o.e.e.NodeEnvironment    ] [node-1] heap size [6.8gb], compressed ordinary object pointers [true]
[2020-05-28T13:18:24,205][INFO ][o.e.n.Node               ] [node-1] node name [node-1], node ID [HUPXjvXWSjmiCa9IY7yaLA], cluster name [htru-webapp]
[2020-05-28T13:18:24,205][INFO ][o.e.n.Node               ] [node-1] version[7.6.2], pid[7176], build[default/zip/ef48eb35cf30adf4db14086e8aabd07ef6fb113f/2020-03-26T06:34:37.794943Z], OS[Windows Server 2012 R2/6.3/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_101/25.101-b13]
[2020-05-28T13:18:24,205][INFO ][o.e.n.Node               ] [node-1] JVM home [C:\Program Files\Java\jdk1.8.0_101\jre]
[2020-05-28T13:18:24,205][INFO ][o.e.n.Node               ] [node-1] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms7g, -Xmx7g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=C:\Users\parsar\AppData\Local\Temp\2\elasticsearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -XX:MaxDirectMemorySize=1610612736, -Delasticsearch, -Des.path.home=Z:\Code for HTRU Tools\Production\Elasticsearch\elasticsearch-7.6.2 - HTRU - A, -Des.path.conf=Z:\Code for HTRU Tools\Production\Elasticsearch\elasticsearch-7.6.2 - HTRU - A\config, -Des.distribution.flavor=default, -Des.distribution.type=zip, -Des.bundled_jdk=true]
[2020-05-28T13:18:35,862][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [aggs-matrix-stats]
[2020-05-28T13:18:35,878][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [analysis-common]
[2020-05-28T13:18:35,878][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [flattened]
[2020-05-28T13:18:35,878][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [frozen-indices]
[2020-05-28T13:18:35,878][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [ingest-common]
[2020-05-28T13:18:35,878][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [ingest-geoip]
[2020-05-28T13:18:35,893][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [ingest-user-agent]
[2020-05-28T13:18:35,893][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [lang-expression]
[2020-05-28T13:18:35,893][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [lang-mustache]
[2020-05-28T13:18:35,893][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [lang-painless]
[2020-05-28T13:18:35,893][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [mapper-extras]
[2020-05-28T13:18:35,908][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [parent-join]
[2020-05-28T13:18:35,908][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [percolator]
[2020-05-28T13:18:35,908][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [rank-eval]
[2020-05-28T13:18:35,908][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [reindex]
[2020-05-28T13:18:35,908][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [repository-url]
[2020-05-28T13:18:35,908][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [search-business-rules]
[2020-05-28T13:18:35,924][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [spatial]
[2020-05-28T13:18:35,924][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [transform]
[2020-05-28T13:18:35,924][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [transport-netty4]
[2020-05-28T13:18:35,924][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [vectors]
[2020-05-28T13:18:35,924][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-analytics]
[2020-05-28T13:18:35,940][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ccr]
[2020-05-28T13:18:35,940][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-core]
[2020-05-28T13:18:35,940][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-deprecation]
[2020-05-28T13:18:35,940][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-enrich]
[2020-05-28T13:18:35,940][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-graph]
[2020-05-28T13:18:35,940][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ilm]
[2020-05-28T13:18:35,955][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-logstash]
[2020-05-28T13:18:35,955][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ml]
[2020-05-28T13:18:35,955][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-monitoring]
[2020-05-28T13:18:35,955][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-rollup]
[2020-05-28T13:18:35,955][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-security]
[2020-05-28T13:18:35,971][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-sql]
[2020-05-28T13:18:35,971][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-voting-only-node]
[2020-05-28T13:18:35,971][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-watcher]
[2020-05-28T13:18:35,971][INFO ][o.e.p.PluginsService     ] [node-1] no plugins loaded
[2020-05-28T13:18:47,518][INFO ][o.e.x.s.a.s.FileRolesStore] [node-1] parsed [0] roles from file [Z:\Code for HTRU Tools\Production\Elasticsearch\elasticsearch-7.6.2 - HTRU - A\config\roles.yml]
[2020-05-28T13:18:49,049][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-1] [controller/7384] [Main.cc@110] controller (64 bit): Version 7.6.2 (Build e06ef9d86d5332) Copyright (c) 2020 Elasticsearch BV
[2020-05-28T13:18:50,799][DEBUG][o.e.a.ActionModule       ] [node-1] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2020-05-28T13:18:51,330][INFO ][o.e.d.DiscoveryModule    ] [node-1] using discovery type [zen] and seed hosts providers [settings]
[2020-05-28T13:18:54,643][INFO ][o.e.n.Node               ] [node-1] initialized
[2020-05-28T13:18:54,643][INFO ][o.e.n.Node               ] [node-1] starting ...
[2020-05-28T13:18:55,846][INFO ][o.e.t.TransportService   ] [node-1] publish_address {10.229.31.92:9300}, bound_addresses {[::]:9300}
[2020-05-28T13:18:57,565][INFO ][o.e.b.BootstrapChecks    ] [node-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-05-28T13:18:57,690][INFO ][o.e.c.c.Coordinator      ] [node-1] cluster UUID [XMMheSowS56cM4YVjRMrbg]
[2020-05-28T13:19:07,737][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered or elected yet, an election requires a node with id [NNKvL-XLTG-2hXF_62jYcA], have discovered [{node-1}{HUPXjvXWSjmiCa9IY7yaLA}{AS43jt9vTN-Vikgt0lQyDA}{10.229.31.92}{10.229.31.92:9300}{dilm}{ml.machine_memory=17179332608, xpack.installed=true, ml.max_open_jobs=20}] which is not a quorum; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, [::1]:9305] from hosts providers and [{node-1}{HUPXjvXWSjmiCa9IY7yaLA}{AS43jt9vTN-Vikgt0lQyDA}{10.229.31.92}{10.229.31.92:9300}{dilm}{ml.machine_memory=17179332608, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 45, last-accepted version 407 in term 45
2

So I start node-2 and this is the startup logs:

[2020-05-28T13:22:39,271][INFO ][o.e.e.NodeEnvironment    ] [node-2] using [1] data paths, mounts [[DATA-E (Z:)]], net usable_space [62.5gb], net total_space [35.9tb], types [NTFS]
[2020-05-28T13:22:39,287][INFO ][o.e.e.NodeEnvironment    ] [node-2] heap size [989.8mb], compressed ordinary object pointers [true]
[2020-05-28T13:22:40,287][INFO ][o.e.n.Node               ] [node-2] node name [node-2], node ID [NNKvL-XLTG-2hXF_62jYcA], cluster name [htru-webapp]
[2020-05-28T13:22:40,287][INFO ][o.e.n.Node               ] [node-2] version[7.6.2], pid[6728], build[default/zip/ef48eb35cf30adf4db14086e8aabd07ef6fb113f/2020-03-26T06:34:37.794943Z], OS[Windows Server 2012 R2/6.3/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_101/25.101-b13]
[2020-05-28T13:22:40,302][INFO ][o.e.n.Node               ] [node-2] JVM home [C:\Program Files\Java\jdk1.8.0_101\jre]
[2020-05-28T13:22:40,302][INFO ][o.e.n.Node               ] [node-2] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=C:\Users\parsar\AppData\Local\Temp\2\elasticsearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -XX:MaxDirectMemorySize=536870912, -Delasticsearch, -Des.path.home=Z:\Code for HTRU Tools\Production\Elasticsearch\elasticsearch-7.6.2 - HTRU - B, -Des.path.conf=Z:\Code for HTRU Tools\Production\Elasticsearch\elasticsearch-7.6.2 - HTRU - B\config, -Des.distribution.flavor=default, -Des.distribution.type=zip, -Des.bundled_jdk=true]
[2020-05-28T13:22:53,506][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [aggs-matrix-stats]
[2020-05-28T13:22:53,506][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [analysis-common]
[2020-05-28T13:22:53,506][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [flattened]
[2020-05-28T13:22:53,521][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [frozen-indices]
[2020-05-28T13:22:53,521][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [ingest-common]
[2020-05-28T13:22:53,521][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [ingest-geoip]
[2020-05-28T13:22:53,521][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [ingest-user-agent]
[2020-05-28T13:22:53,521][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [lang-expression]
[2020-05-28T13:22:53,521][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [lang-mustache]
[2020-05-28T13:22:53,537][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [lang-painless]
[2020-05-28T13:22:53,537][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [mapper-extras]
[2020-05-28T13:22:53,537][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [parent-join]
[2020-05-28T13:22:53,537][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [percolator]
[2020-05-28T13:22:53,537][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [rank-eval]
[2020-05-28T13:22:53,537][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [reindex]
[2020-05-28T13:22:53,537][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [repository-url]
[2020-05-28T13:22:53,552][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [search-business-rules]
[2020-05-28T13:22:53,552][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [spatial]
[2020-05-28T13:22:53,552][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [transform]
[2020-05-28T13:22:53,552][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [transport-netty4]
[2020-05-28T13:22:53,552][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [vectors]
[2020-05-28T13:22:53,552][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-analytics]
[2020-05-28T13:22:53,568][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-ccr]
[2020-05-28T13:22:53,568][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-core]
[2020-05-28T13:22:53,568][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-deprecation]
[2020-05-28T13:22:53,568][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-enrich]
[2020-05-28T13:22:53,568][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-graph]
[2020-05-28T13:22:53,568][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-ilm]
[2020-05-28T13:22:53,584][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-logstash]
[2020-05-28T13:22:53,584][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-ml]
[2020-05-28T13:22:53,584][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-monitoring]
[2020-05-28T13:22:53,584][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-rollup]
[2020-05-28T13:22:53,584][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-security]
[2020-05-28T13:22:53,584][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-sql]
[2020-05-28T13:22:53,599][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-voting-only-node]
[2020-05-28T13:22:53,599][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-watcher]
[2020-05-28T13:22:53,599][INFO ][o.e.p.PluginsService     ] [node-2] no plugins loaded
[2020-05-28T13:23:04,521][INFO ][o.e.x.s.a.s.FileRolesStore] [node-2] parsed [0] roles from file [Z:\Code for HTRU Tools\Production\Elasticsearch\elasticsearch-7.6.2 - HTRU - B\config\roles.yml]
[2020-05-28T13:23:06,131][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-2] [controller/4880] [Main.cc@110] controller (64 bit): Version 7.6.2 (Build e06ef9d86d5332) Copyright (c) 2020 Elasticsearch BV
[2020-05-28T13:23:07,552][DEBUG][o.e.a.ActionModule       ] [node-2] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2020-05-28T13:23:08,006][INFO ][o.e.d.DiscoveryModule    ] [node-2] using discovery type [zen] and seed hosts providers [settings]
[2020-05-28T13:23:10,724][INFO ][o.e.n.Node               ] [node-2] initialized
[2020-05-28T13:23:10,724][INFO ][o.e.n.Node               ] [node-2] starting ...
[2020-05-28T13:23:11,427][INFO ][o.e.t.TransportService   ] [node-2] publish_address {10.229.31.92:9300}, bound_addresses {[::]:9300}
[2020-05-28T13:23:12,709][INFO ][o.e.b.BootstrapChecks    ] [node-2] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-05-28T13:23:12,834][INFO ][o.e.c.c.Coordinator      ] [node-2] cluster UUID [XMMheSowS56cM4YVjRMrbg]
[2020-05-28T13:23:13,209][INFO ][o.e.c.s.MasterService    ] [node-2] elected-as-master ([1] nodes joined)[{node-2}{NNKvL-XLTG-2hXF_62jYcA}{XKuWSiO9SY6KFQoSG0WRzw}{10.229.31.92}{10.229.31.92:9300}{dilm}{ml.machine_memory=17179332608, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 46, version: 408, delta: master node changed {previous [], current [{node-2}{NNKvL-XLTG-2hXF_62jYcA}{XKuWSiO9SY6KFQoSG0WRzw}{10.229.31.92}{10.229.31.92:9300}{dilm}{ml.machine_memory=17179332608, xpack.installed=true, ml.max_open_jobs=20}]}
[2020-05-28T13:23:13,662][INFO ][o.e.c.s.ClusterApplierService] [node-2] master node changed {previous [], current [{node-2}{NNKvL-XLTG-2hXF_62jYcA}{XKuWSiO9SY6KFQoSG0WRzw}{10.229.31.92}{10.229.31.92:9300}{dilm}{ml.machine_memory=17179332608, xpack.installed=true, ml.max_open_jobs=20}]}, term: 46, version: 408, reason: Publication{term=46, version=408}
[2020-05-28T13:23:13,927][INFO ][o.e.h.AbstractHttpServerTransport] [node-2] publish_address {10.229.31.92:9378}, bound_addresses {[::]:9378}
[2020-05-28T13:23:13,943][INFO ][o.e.n.Node               ] [node-2] started
[2020-05-28T13:23:14,521][INFO ][o.e.l.LicenseService     ] [node-2] license [00371069-a67b-4394-b069-74ffc1738d5e] mode [basic] - valid
[2020-05-28T13:23:14,537][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [node-2] Active license is now [BASIC]; Security is disabled
[2020-05-28T13:23:14,552][INFO ][o.e.g.GatewayService     ] [node-2] recovered [1] indices into cluster_state
[2020-05-28T13:23:20,224][INFO ][o.e.c.r.a.AllocationService] [node-2] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[htru-docs][0]]]).
[2020-05-28T13:24:13,977][INFO ][o.e.c.r.a.DiskThresholdMonitor] [node-2] low disk watermark [100gb] exceeded on [NNKvL-XLTG-2hXF_62jYcA][node-2][Z:\Code for HTRU Tools\Production\Elasticsearch\data\node-2\nodes\0] free: 62.5gb[0.1%], replicas will not be assigned to this node
[2020-05-28T13:25:00,024][INFO ][o.e.c.s.MasterService    ] [node-2] node-join[{node-1}{HUPXjvXWSjmiCa9IY7yaLA}{dccSqRU7TReqahBjq50gBg}{10.229.31.92}{10.229.31.92:9301}{dilm}{ml.machine_memory=17179332608, ml.max_open_jobs=20, xpack.installed=true} join existing leader], term: 46, version: 413, delta: added {{node-1}{HUPXjvXWSjmiCa9IY7yaLA}{dccSqRU7TReqahBjq50gBg}{10.229.31.92}{10.229.31.92:9301}{dilm}{ml.machine_memory=17179332608, ml.max_open_jobs=20, xpack.installed=true}}
[2020-05-28T13:25:01,961][INFO ][o.e.c.s.ClusterApplierService] [node-2] added {{node-1}{HUPXjvXWSjmiCa9IY7yaLA}{dccSqRU7TReqahBjq50gBg}{10.229.31.92}{10.229.31.92:9301}{dilm}{ml.machine_memory=17179332608, ml.max_open_jobs=20, xpack.installed=true}}, term: 46, version: 413, reason: Publication{term=46, version=413}
[2020-05-28T13:25:02,040][INFO ][o.e.c.r.a.DiskThresholdMonitor] [node-2] low disk watermark [100gb] exceeded on [HUPXjvXWSjmiCa9IY7yaLA][node-1][Z:\Code for HTRU Tools\Production\Elasticsearch\elasticsearch-7.6.2 - HTRU - A\data\nodes\0] free: 62.5gb[0.1%], replicas will not be assigned to this node
3

My cluster health remains yellow and only about half of my index "store" is loaded. Can anyone tell me what might cause Elastic to flip like this and designate my node-2 as the initial master, and why only half of my cluster is loading? I see it says the low disk watermark is exceeded, but I have not changed any settings and the server still has disk space so I'm not sure why this error is being thrown all of a sudden. Here are the configs for both.

node-1

# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: htru-webapp
cluster.routing.allocation.disk.watermark.low: 100gb
cluster.routing.allocation.disk.watermark.high: 50gb
cluster.routing.allocation.disk.watermark.flood_stage: 10gb
cluster.info.update.interval: 1m
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1

# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9377
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["node-1"]

node-2

# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: htru-webapp
cluster.routing.allocation.disk.watermark.low: 100gb
cluster.routing.allocation.disk.watermark.high: 50gb
cluster.routing.allocation.disk.watermark.flood_stage: 10gb
cluster.info.update.interval: 1m
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-2
#
# Add custom attributes to the node:
#
node.data: true
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: Z:\Code for HTRU Tools\Production\Elasticsearch\data\node-2
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9378
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["node-1"]

Please any help or guidance would be greatly appreciated. Thank you in advance.

4

You designate nodes as being master-eligible and which one that is acting as the master can change at any time. This is part of the design and not a problem. You should however be aware that you need at least 3 master-eligible nodes in a cluster for it to be highly available and be able to continue operating even if one node is down. This is due to the fact that an absolute majority is required to elect a master and the majority of two is two.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.