I have 3 elasticsearch nodes, nodes 1 and 2 (Red Hat Enterprise Linux release 8.6 ) are configured as master, I try to start the service and it only starts node 3 which is data only, nodes 1 and 2 show the following in the log:
systemctl start elasticsearch
Job for elasticsearch.service failed because a timeout was exceeded.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
[2023-01-25T16:39:18,326][WARN ][stderr ] [srielk11.sri.ad] Jan 25, 2023 4:39:18 PM org.apache.lucene.store.MMapDirectory lookupProvider
[2023-01-25T16:39:18,327][WARN ][stderr ] [srielk11.sri.ad] WARNING: You are running with Java 19. To make full use of MMapDirectory, please pass '--enable-preview' to the Java command line.
[2023-01-25T16:39:18,336][INFO ][o.e.e.NodeEnvironment ] [srielk11.sri.ad] using [1] data paths, mounts [[/var (/dev/mapper/rhel-var)]], net usable_space [18.9gb], net total_space [479.7gb], types [xfs]
[2023-01-25T16:39:18,337][INFO ][o.e.e.NodeEnvironment ] [srielk11.sri.ad] heap size [18gb], compressed ordinary object pointers [true]
[2023-01-25T16:39:18,415][INFO ][o.e.n.Node ] [srielk11.sri.ad] node name [srielk11.sri.ad], node ID [l8hSG5V0TpCkxKZi8Aj_YA], cluster name [sriclusterbdd], roles [master, data]
[2023-01-25T16:39:21,384][INFO ][o.e.x.s.Security ] [srielk11.sri.ad] Security is enabled
[2023-01-25T16:39:21,776][INFO ][o.e.x.s.a.s.FileRolesStore] [srielk11.sri.ad] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2023-01-25T16:39:22,523][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [srielk11.sri.ad] [controller/56068] [Main.cc@123] controller (64 bit): Version 8.5.0 (Build 3922fab346e761) Copyright (c) 2022 Elasticsearch BV
[2023-01-25T16:39:23,147][INFO ][o.e.t.n.NettyAllocator ] [srielk11.sri.ad] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=16mb}]
[2023-01-25T16:39:23,175][INFO ][o.e.i.r.RecoverySettings ] [srielk11.sri.ad] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2023-01-25T16:39:23,217][INFO ][o.e.d.DiscoveryModule ] [srielk11.sri.ad] using discovery type [multi-node] and seed hosts providers [settings]
[2023-01-25T16:39:24,483][INFO ][o.e.n.Node ] [srielk11.sri.ad] initialized
[2023-01-25T16:39:24,484][INFO ][o.e.n.Node ] [srielk11.sri.ad] starting ...
[2023-01-25T16:39:24,516][INFO ][o.e.x.s.c.f.PersistentCache] [srielk11.sri.ad] persistent cache index loaded
[2023-01-25T16:39:24,517][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [srielk11.sri.ad] deprecation component started
[2023-01-25T16:39:24,638][INFO ][o.e.t.TransportService ] [srielk11.sri.ad] publish_address {10.2.2.209:9300}, bound_addresses {10.2.2.209:9300}
[2023-01-25T16:39:25,693][INFO ][o.e.b.BootstrapChecks ] [srielk11.sri.ad] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2023-01-25T16:39:25,715][INFO ][o.e.c.c.ClusterBootstrapService] [srielk11.sri.ad] this node is locked into cluster UUID [L6nMXzvoQmqrVEwFoe-FfA] and will not attempt further cluster bootstrapping
[2023-01-25T16:39:35,726][WARN ][o.e.c.c.ClusterFormationFailureHelper] [srielk11.sri.ad] master not discovered or elected yet, an election requires at least 2 nodes with ids from [HXuwVaLZTJCm7FfvKbfUQw, l8hSG5V0TpCkxKZi8Aj_YA, DS1LTtR1RZWoVh1VUqUI9A], have only discovered non-quorum [{srielk11.sri.ad}{l8hSG5V0TpCkxKZi8Aj_YA}{p0yzu1n0T5S9-dhZGJymiQ}{srielk11.sri.ad}{10.2.2.209}{10.2.2.209:9300}{dm}]; discovery will continue using [10.2.2.210:9300, 10.2.2.102:9300] from hosts providers and [{srielk11.sri.ad}{l8hSG5V0TpCkxKZi8Aj_YA}{p0yzu1n0T5S9-dhZGJymiQ}{srielk11.sri.ad}{10.2.2.209}{10.2.2.209:9300}{dm}] from last-known cluster state; node term 150, last-accepted version 14585 in term 150
[2023-01-25T16:39:45,728][WARN ][o.e.c.c.ClusterFormationFailureHelper] [srielk11.sri.ad] master not discovered or elected yet, an election requires at least 2 nodes with ids from [HXuwVaLZTJCm7FfvKbfUQw, l8hSG5V0TpCkxKZi8Aj_YA, DS1LTtR1RZWoVh1VUqUI9A], have only discovered non-quorum [{srielk11.sri.ad}{l8hSG5V0TpCkxKZi8Aj_YA}{p0yzu1n0T5S9-dhZGJymiQ}{srielk11.sri.ad}{10.2.2.209}{10.2.2.209:9300}{dm}]; discovery will continue using [10.2.2.210:9300, 10.2.2.102:9300] from hosts providers and [{srielk11.sri.ad}{l8hSG5V0TpCkxKZi8Aj_YA}{p0yzu1n0T5S9-dhZGJymiQ}{srielk11.sri.ad}{10.2.2.209}{10.2.2.209:9300}{dm}] from last-known cluster state; node term 150, last-accepted version 14585 in term 150
[2023-01-25T16:39:55,729][WARN ][o.e.c.c.ClusterFormationFailureHelper] [srielk11.sri.ad] master not discovered or elected yet, an election requires at least 2 nodes with ids from [HXuwVaLZTJCm7FfvKbfUQw, l8hSG5V0TpCkxKZi8Aj_YA, DS1LTtR1RZWoVh1VUqUI9A], have only discovered non-quorum [{srielk11.sri.ad}{l8hSG5V0TpCkxKZi8Aj_YA}{p0yzu1n0T5S9-dhZGJymiQ}{srielk11.sri.ad}{10.2.2.209}{10.2.2.209:9300}{dm}]; discovery will continue using [10.2.2.210:9300, 10.2.2.102:9300] from hosts providers and [{srielk11.sri.ad}{l8hSG5V0TpCkxKZi8Aj_YA}{p0yzu1n0T5S9-dhZGJymiQ}{srielk11.sri.ad}{10.2.2.209}{10.2.2.209:9300}{dm}] from last-known cluster state; node term 150, last-accepted version 14585 in term 150
[2023-01-25T16:39:55,730][WARN ][o.e.n.Node ] [srielk11.sri.ad] timed out while waiting for initial discovery state - timeout: 30s
[2023-01-25T16:39:55,734][INFO ][o.e.h.AbstractHttpServerTransport] [srielk11.sri.ad] publish_address {10.2.2.209:9200}, bound_addresses {10.2.2.209:9200}
[2023-01-25T16:39:55,734][INFO ][o.e.n.Node ] [srielk11.sri.ad] started {srielk11.sri.ad}{l8hSG5V0TpCkxKZi8Aj_YA}{p0yzu1n0T5S9-dhZGJymiQ}{srielk11.sri.ad}{10.2.2.209}{10.2.2.209:9300}{dm}{xpack.installed=true}
[2023-01-25T16:40:05,731][WARN ][o.e.c.c.ClusterFormationFailureHelper] [srielk11.sri.ad] master not discovered or elected yet, an election requires at least 2 nodes with ids from [HXuwVaLZTJCm7FfvKbfUQw, l8hSG5V0TpCkxKZi8Aj_YA, DS1LTtR1RZWoVh1VUqUI9A], have only discovered non-quorum [{srielk11.sri.ad}{l8hSG5V0TpCkxKZi8Aj_YA}{p0yzu1n0T5S9-dhZGJymiQ}{srielk11.sri.ad}{10.2.2.209}{10.2.2.209:9300}{dm}]; discovery will continue using [10.2.2.210:9300, 10.2.2.102:9300] from hosts providers and [{srielk11.sri.ad}{l8hSG5V0TpCkxKZi8Aj_YA}{p0yzu1n0T5S9-dhZGJymiQ}{srielk11.sri.ad}{10.2.2.209}{10.2.2.209:9300}{dm}] from last-known cluster state; node term 150, last-accepted version 14585 in term 150
[2023-01-25T16:40:15,732][WARN ][o.e.c.c.ClusterFormationFailureHelper] [srielk11.sri.ad] master not discovered or elected yet, an election requires at least 2 nodes with ids from [HXuwVaLZTJCm7FfvKbfUQw, l8hSG5V0TpCkxKZi8Aj_YA, DS1LTtR1RZWoVh1VUqUI9A], have only discovered non-quorum [{srielk11.sri.ad}{l8hSG5V0TpCkxKZi8Aj_YA}{p0yzu1n0T5S9-dhZGJymiQ}{srielk11.sri.ad}{10.2.2.209}{10.2.2.209:9300}{dm}]; discovery will continue using [10.2.2.210:9300, 10.2.2.102:9300] from hosts providers and [{srielk11.sri.ad}{l8hSG5V0TpCkxKZi8Aj_YA}{p0yzu1n0T5S9-dhZGJymiQ}{srielk11.sri.ad}{10.2.2.209}{10.2.2.209:9300}{dm}] from last-known cluster state; node term 150, last-accepted version 14585 in term 150
[2023-01-25T16:40:25,733][WARN ][o.e.c.c.ClusterFormationFailureHelper] [srielk11.sri.ad] master not discovered or elected yet, an election requires at least 2 nodes with ids from [HXuwVaLZTJCm7FfvKbfUQw, l8hSG5V0TpCkxKZi8Aj_YA, DS1LTtR1RZWoVh1VUqUI9A], have only discovered non-quorum [{srielk11.sri.ad}{l8hSG5V0TpCkxKZi8Aj_YA}{p0yzu1n0T5S9-dhZGJymiQ}{srielk11.sri.ad}{10.2.2.209}{10.2.2.209:9300}{dm}]; discovery will continue using [10.2.2.210:9300, 10.2.2.102:9300] from hosts providers and [{srielk11.sri.ad}{l8hSG5V0TpCkxKZi8Aj_YA}{p0yzu1n0T5S9-dhZGJymiQ}{srielk11.sri.ad}{10.2.2.209}{10.2.2.209:9300}{dm}] from last-known cluster state; node term 150, last-accepted version 14585 in term 150
[2023-01-25T16:40:35,735][WARN ][o.e.c.c.ClusterFormationFailureHelper] [srielk11.sri.ad] master not discovered or elected yet, an election requires at least 2 nodes with ids from [HXuwVaLZTJCm7FfvKbfUQw, l8hSG5V0TpCkxKZi8Aj_YA, DS1LTtR1RZWoVh1VUqUI9A], have only discovered non-quorum [{srielk11.sri.ad}{l8hSG5V0TpCkxKZi8Aj_YA}{p0yzu1n0T5S9-dhZGJymiQ}{srielk11.sri.ad}{10.2.2.209}{10.2.2.209:9300}{dm}]; discovery will continue using [10.2.2.210:9300, 10.2.2.102:9300] from hosts providers and [{srielk11.sri.ad}{l8hSG5V0TpCkxKZi8Aj_YA}{p0yzu1n0T5S9-dhZGJymiQ}{srielk11.sri.ad}{10.2.2.209}{10.2.2.209:9300}{dm}] from last-known cluster state; node term 150, last-accepted version 14585 in term 150
[2023-01-25T16:40:36,530][INFO ][o.e.n.Node ] [srielk11.sri.ad] stopping ...
[2023-01-25T16:40:36,533][INFO ][o.e.x.w.WatcherService ] [srielk11.sri.ad] stopping watch service, reason [shutdown initiated]
[2023-01-25T16:40:36,534][INFO ][o.e.x.w.WatcherLifeCycleService] [srielk11.sri.ad] watcher has stopped and shutdown
[2023-01-25T16:40:37,014][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [srielk11.sri.ad] [controller/56068] [Main.cc@176] ML controller exiting
[2023-01-25T16:40:37,015][INFO ][o.e.x.m.p.NativeController] [srielk11.sri.ad] Native controller process has stopped - no new native processes can be started
[2023-01-25T16:40:37,016][WARN ][stderr ] [srielk11.sri.ad] The system environment variables are not available to Log4j due to security restrictions: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getenv.*")The system environment variables are not available to Log4j due to security restrictions: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getenv.*")The system environment variables are not available to Log4j due to security restrictions: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getenv.*")The system environment variables are not available to Log4j due to security restrictions: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getenv.*")The system environment variables are not available to Log4j due to security restrictions: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getenv.*")
[2023-01-25T16:40:37,017][WARN ][stderr ] [srielk11.sri.ad] java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getenv.*")
[2023-01-25T16:40:37,017][WARN ][stderr ] [srielk11.sri.ad] at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:485)
[2023-01-25T16:40:37,018][WARN ][stderr ] [srielk11.sri.ad] at java.base/java.security.AccessController.checkPermission(AccessController.java:1068)
[2023-01-25T16:40:37,018][WARN ][stderr ] [srielk11.sri.ad] at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411)
[2023-01-25T16:40:37,018][WARN ][stderr ] [srielk11.sri.ad] at java.base/java.lang.System.getenv(System.java:1196)
[2023-01-25T16:40:37,018][WARN ][stderr ] [srielk11.sri.ad] at org.apache.logging.log4j@2.18.0/org.apache.logging.log4j.util.EnvironmentPropertySource.containsProperty(EnvironmentPropertySource.java:99)
[2023-01-25T16:40:37,018][WARN ][stderr ] [srielk11.sri.ad] at org.apache.logging.log4j@2.18.0/org.apache.logging.log4j.util.PropertiesUtil$Environment.get(PropertiesUtil.java:513)
[2023-01-25T16:40:37,018][WARN ][stderr ] [srielk11.sri.ad] at org.apache.logging.log4j@2.18.0/org.apache.logging.log4j.util.PropertiesUtil$Environment.access$200(PropertiesUtil.java:434)
[2023-01-25T16:40:37,018][WARN ][stderr ] [srielk11.sri.ad] at org.apache.logging.log4j@2.18.0/org.apache.logging.log4j.util.PropertiesUtil.getStringProperty(PropertiesUtil.java:382)
[2023-01-25T16:40:37,018][WARN ][stderr ] [srielk11.sri.ad] at org.apache.logging.log4j@2.18.0/org.apache.logging.log4j.util.PropertiesUtil.getBooleanProperty(PropertiesUtil.java:169)
[2023-01-25T16:40:37,018][WARN ][stderr ] [srielk11.sri.ad] at org.apache.logging.log4j@2.18.0/org.apache.logging.log4j.status.StatusLogger.isDebugPropertyEnabled(StatusLogger.java:143)
[2023-01-25T16:40:37,018][WARN ][stderr ] [srielk11.sri.ad] at org.apache.logging.log4j@2.18.0/org.apache.logging.log4j.status.StatusLogger.isEnabled(StatusLogger.java:426)
[2023-01-25T16:40:37,018][WARN ][stderr ] [srielk11.sri.ad] at org.apache.logging.log4j@2.18.0/org.apache.logging.log4j.status.StatusLogger.isEnabled(StatusLogger.java:327)
[2023-01-25T16:40:37,018][WARN ][stderr ] [srielk11.sri.ad] at org.apache.logging.log4j@2.18.0/org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1982)
[2023-01-25T16:40:37,018][WARN ][stderr ] [srielk11.sri.ad] at org.apache.logging.log4j@2.18.0/org.apache.logging.log4j.spi.AbstractLogger.error(AbstractLogger.java:740)
[2023-01-25T16:40:37,019][WARN ][stderr ] [srielk11.sri.ad] at org.apache.logging.log4j.core@8.5.0/org.apache.logging.log4j.core.appender.DefaultErrorHandler.error(DefaultErrorHandler.java:61)
[2023-01-25T16:40:37,088][WARN ][stderr ] [srielk11.sri.ad] The system environment variables are not available to Log4j due to security restrictions: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getenv.*")
[2023-01-25T16:40:37,088][WARN ][stderr ] [srielk11.sri.ad] java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getenv.*")
[2023-01-25T16:40:37,016][WARN ][stderr ] [srielk11.sri.ad] The system environment variables are not available to Log4j due to security restrictions: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getenv.*")The system environment variables are not available to Log4j due to security restrictions: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getenv.*")
[2023-01-25T16:40:37,016][WARN ][stderr ] [srielk11.sri.ad] The system environment variables are not available to Log4j due to security restrictions: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getenv.*")
[2023-01-25T16:40:37,015][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [srielk11.sri.ad] uncaught exception in thread [process reaper (pid 56068)]
java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "modifyThread")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:485) ~[?:?]
at java.security.AccessController.checkPermission(AccessController.java:1068) ~[?:?]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]
at org.elasticsearch.secure_sm.SecureSM.checkThreadAccess(SecureSM.java:166) ~[?:?]
at org.elasticsearch.secure_sm.SecureSM.checkAccess(SecureSM.java:120) ~[?:?]
at java.lang.Thread.checkAccess(Thread.java:2360) ~[?:?]
at java.lang.Thread.setDaemon(Thread.java:2308) ~[?:?]
at java.lang.ProcessHandleImpl.lambda$static$0(ProcessHandleImpl.java:103) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:637) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:928) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1021) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1158) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
at jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:186) ~[?:?]