Kibana server is not ready yet (license: null)

I am attempting to set up an Elasticsearch cluster consisting of three master-eligible nodes.

elasticsearch.yml looks like this on all of the nodes:

cluster.name: elkdev
node.name: {machinename}.dev.company.local
network.host: 0.0.0.0
http.port: 9200
node.master: false
node.data: true
node.ingest: false
discovery.seed_hosts: ["{node1name}.dev.company.local", "{node2name}.dev.company.local", "{node3name}.dev.company.local"]
cluster.initial_master_nodes: ["{node1name}.dev.company.local"]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

So, they all have the fully qualified host name as node.name, all of the fully qualified host names are included in discovery.seed_hosts, and the fully qualified host name for the FIRST node is contained in cluster.initial_master_nodes for all of the nodes.

If I run curl http://{node1name}.dev.company.local:9200/_xpack?pretty, it looks like it is started up and running successfully. Maybe importantly, it contains license information:

  "license" : {
    "uid" : "8dd0486a-9f55-4c92-8711-2c32336a746c",
    "type" : "basic",
    "mode" : "basic",
    "status" : "active"
  },

If I run the same command for either of the other nodes, I get "license": null in the output.

On nodes 2 & 3, I see this error in the log:

[2021-09-07T09:54:08,288][WARN ][r.suppressed             ] [{machinename}.dev.company.local] path: /logstash, params: {index=logstash}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) [elasticsearch-7.9.0.jar:7.9.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]

If I run journalctl -u kibana.service, I see this kind of thing in the output:

Sep 07 09:58:45 {machinename}.dev.company.local kibana[121433]: {"type":"log","@timestamp":"2021-09-07T14:58:45Z","tags":["warning","savedobjects-service"],"pid":121433,"message":"Unable to connect to Elasticsearch. Error: Request Timeout after 30000ms"}

I've been searching this forum and elsewhere for similar posts for a few days, and so far have not found a suggestion that works. Anyone have any ideas for further debugging?

Cheers,
Joe

That's the reason. Is there more to the logs from that node?

This is from the same node (node 2).

In here, 10.xxx.xxx.xx1 is the IP of node 1 (the one that seems to be working), 10.xxx.xxx.xx2 is the IP of node 2 (the node that the log is from), and 10.xxx.xxx.xx3 is the IP of node 3 (the one that is behaving the same as node 2).

[2021-09-08T09:09:26,580][WARN ][r.suppressed             ] [node2name.dev.company.locall] path: /_license, params: {}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) [elasticsearch-7.9.0.jar:7.9.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-08T09:09:26,652][WARN ][r.suppressed             ] [node2name.dev.company.local] path: /_template/logstash, params: {name=logstash}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) [elasticsearch-7.9.0.jar:7.9.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-08T09:09:29,505][WARN ][r.suppressed             ] [node2name.dev.company.local] path: /.kibana_task_manager, params: {index=.kibana_task_manager}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) [elasticsearch-7.9.0.jar:7.9.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-08T09:09:30,925][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{DxzxkKlNRFeH4yCZA_ZYbw}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dlrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-08T09:09:40,926][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{DxzxkKlNRFeH4yCZA_ZYbw}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dlrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-08T09:09:50,328][WARN ][r.suppressed             ] [node2name.dev.company.local] path: /.kibana, params: {index=.kibana}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) [elasticsearch-7.9.0.jar:7.9.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-08T09:09:50,927][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{DxzxkKlNRFeH4yCZA_ZYbw}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dlrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-08T09:09:56,628][WARN ][r.suppressed             ] [node2name.dev.company.local] path: /_license, params: {}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) [elasticsearch-7.9.0.jar:7.9.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-08T09:09:56,631][WARN ][r.suppressed             ] [node2name.dev.company.local] path: /_template/logstash, params: {name=logstash}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) [elasticsearch-7.9.0.jar:7.9.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-08T09:09:56,684][WARN ][r.suppressed             ] [node2name.dev.company.local] path: /logstash, params: {index=logstash}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) [elasticsearch-7.9.0.jar:7.9.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-08T09:10:00,928][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{DxzxkKlNRFeH4yCZA_ZYbw}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dlrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-08T09:10:02,008][WARN ][r.suppressed             ] [node2name.dev.company.local] path: /.kibana_task_manager, params: {index=.kibana_task_manager}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) [elasticsearch-7.9.0.jar:7.9.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-08T09:10:10,928][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{DxzxkKlNRFeH4yCZA_ZYbw}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dlrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0

And here is the beginning of the log up to when the errors start happening:

[2021-09-08T10:17:56,905][INFO ][o.e.n.Node               ] [node2name.dev.company.local] version[7.9.0], pid[200627], build[default/rpm/a479a2a7fce0389512d6a9361301708b92dff667/2020-08-11T21:36:48.204330Z], OS[Linux/3.10.0-1160.36.2.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/14.0.1/14.0.1+7]
[2021-09-08T10:17:56,909][INFO ][o.e.n.Node               ] [node2name.dev.company.local] JVM home [/usr/share/elasticsearch/jdk]
[2021-09-08T10:17:56,909][INFO ][o.e.n.Node               ] [node2name.dev.company.local] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-10713857089481507232, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2021-09-08T10:17:59,091][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [aggs-matrix-stats]
[2021-09-08T10:17:59,091][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [analysis-common]
[2021-09-08T10:17:59,092][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [constant-keyword]
[2021-09-08T10:17:59,092][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [flattened]
[2021-09-08T10:17:59,092][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [frozen-indices]
[2021-09-08T10:17:59,092][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [ingest-common]
[2021-09-08T10:17:59,093][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [ingest-geoip]
[2021-09-08T10:17:59,093][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [ingest-user-agent]
[2021-09-08T10:17:59,093][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [kibana]
[2021-09-08T10:17:59,093][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [lang-expression]
[2021-09-08T10:17:59,094][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [lang-mustache]
[2021-09-08T10:17:59,094][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [lang-painless]
[2021-09-08T10:17:59,094][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [mapper-extras]
[2021-09-08T10:17:59,094][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [parent-join]
[2021-09-08T10:17:59,094][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [percolator]
[2021-09-08T10:17:59,095][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [rank-eval]
[2021-09-08T10:17:59,095][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [reindex]
[2021-09-08T10:17:59,095][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [repository-url]
[2021-09-08T10:17:59,095][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [search-business-rules]
[2021-09-08T10:17:59,095][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [searchable-snapshots]
[2021-09-08T10:17:59,095][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [spatial]
[2021-09-08T10:17:59,096][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [systemd]
[2021-09-08T10:17:59,096][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [tasks]
[2021-09-08T10:17:59,096][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [transform]
[2021-09-08T10:17:59,096][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [transport-netty4]
[2021-09-08T10:17:59,096][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [vectors]
[2021-09-08T10:17:59,096][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [wildcard]
[2021-09-08T10:17:59,097][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-analytics]
[2021-09-08T10:17:59,097][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-async]
[2021-09-08T10:17:59,097][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-async-search]
[2021-09-08T10:17:59,097][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-autoscaling]
[2021-09-08T10:17:59,098][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-ccr]
[2021-09-08T10:17:59,098][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-core]
[2021-09-08T10:17:59,098][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-data-streams]
[2021-09-08T10:17:59,098][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-deprecation]
[2021-09-08T10:17:59,098][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-enrich]
[2021-09-08T10:17:59,098][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-eql]
[2021-09-08T10:17:59,099][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-graph]
[2021-09-08T10:17:59,099][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-identity-provider]
[2021-09-08T10:17:59,099][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-ilm]
[2021-09-08T10:17:59,099][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-logstash]
[2021-09-08T10:17:59,099][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-ml]
[2021-09-08T10:17:59,100][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-monitoring]
[2021-09-08T10:17:59,100][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-ql]
[2021-09-08T10:17:59,100][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-rollup]
[2021-09-08T10:17:59,100][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-security]
[2021-09-08T10:17:59,100][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-sql]
[2021-09-08T10:17:59,101][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-stack]
[2021-09-08T10:17:59,101][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-voting-only-node]
[2021-09-08T10:17:59,101][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-watcher]
[2021-09-08T10:17:59,102][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] no plugins loaded
[2021-09-08T10:17:59,154][INFO ][o.e.e.NodeEnvironment    ] [node2name.dev.company.local] using [1] data paths, mounts [[/ (/dev/mapper/centos-root)]], net usable_space [43.3gb], net total_space [49.9gb], types [xfs]
[2021-09-08T10:17:59,154][INFO ][o.e.e.NodeEnvironment    ] [node2name.dev.company.local] heap size [1gb], compressed ordinary object pointers [true]
[2021-09-08T10:17:59,230][INFO ][o.e.n.Node               ] [node2name.dev.company.local] node name [node2name.dev.company.local], node ID [z0BPC1qZS0218vDUDIFpHg], cluster name [elkdev]
[2021-09-08T10:18:02,695][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node2name.dev.company.local] [controller/200855] [Main.cc@114] controller (64 bit): Version 7.9.0 (Build 2639177a4c3ad6) Copyright (c) 2020 Elasticsearch BV
[2021-09-08T10:18:03,530][INFO ][o.e.x.s.a.s.FileRolesStore] [node2name.dev.company.local] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2021-09-08T10:18:04,627][INFO ][o.e.d.DiscoveryModule    ] [node2name.dev.company.local] using discovery type [zen] and seed hosts providers [settings]
[2021-09-08T10:18:05,085][WARN ][o.e.g.DanglingIndicesState] [node2name.dev.company.local] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-09-08T10:18:05,485][INFO ][o.e.n.Node               ] [node2name.dev.company.local] initialized
[2021-09-08T10:18:05,486][INFO ][o.e.n.Node               ] [node2name.dev.company.local] starting ...
[2021-09-08T10:18:05,649][INFO ][o.e.t.TransportService   ] [node2name.dev.company.local] publish_address {10.xxx.xxx.xx2:9300}, bound_addresses {[::]:9300}
[2021-09-08T10:18:05,813][INFO ][o.e.b.BootstrapChecks    ] [node2name.dev.company.local] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2021-09-08T10:18:15,827][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{BUP6LidrQcypP_WjQyAkZQ}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dlrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-08T10:18:25,830][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{BUP6LidrQcypP_WjQyAkZQ}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dlrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-08T10:18:35,831][WARN ][o.e.n.Node               ] [node2name.dev.company.local] timed out while waiting for initial discovery state - timeout: 30s
[2021-09-08T10:18:35,833][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{BUP6LidrQcypP_WjQyAkZQ}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dlrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-08T10:18:35,848][INFO ][o.e.h.AbstractHttpServerTransport] [node2name.dev.company.local] publish_address {10.xxx.xxx.xx2:9200}, bound_addresses {[::]:9200}
[2021-09-08T10:18:35,849][INFO ][o.e.n.Node               ] [node2name.dev.company.local] started

Oh, correction to my original post:

The first node is configured with this:

node.master: true
node.data: false

The other two nodes are configured as mentioned in the original post.

There aren't all currently set up with node.master: true because I thought that might be causing an issue.

Should they all have both node.master: true and node.data: true if I want them all to be master-eligible data nodes?

I tried a couple of things, including setting

node.master: true
node.data: true
node.ingest: true

On all of the nodes, but still no luck.

I made a change to the configuration to add all of the nodes to the initial master nodes list:

cluster.name: elkdev
node.name: node2name.dev.company.local
network.host: 0.0.0.0
http.port: 9200
node.master: true
node.data: true
node.ingest: true
discovery.seed_hosts: ["node1name.dev.company.local", "node2name.dev.company.local", "node3name.dev.company.local"]
cluster.initial_master_nodes: ["node1name.dev.company.local", "node2name.dev.company.local", "node3name.dev.company.local"]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

Still getting the null exceptions, here is the log from node 2 leading up to the first exception:

[2021-09-09T09:46:40,108][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [transform]
[2021-09-09T09:46:40,109][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [transport-netty4]
[2021-09-09T09:46:40,109][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [vectors]
[2021-09-09T09:46:40,109][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [wildcard]
[2021-09-09T09:46:40,109][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-analytics]
[2021-09-09T09:46:40,109][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-async]
[2021-09-09T09:46:40,109][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-async-search]
[2021-09-09T09:46:40,110][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-autoscaling]
[2021-09-09T09:46:40,110][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-ccr]
[2021-09-09T09:46:40,110][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-core]
[2021-09-09T09:46:40,110][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-data-streams]
[2021-09-09T09:46:40,110][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-deprecation]
[2021-09-09T09:46:40,110][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-enrich]
[2021-09-09T09:46:40,110][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-eql]
[2021-09-09T09:46:40,111][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-graph]
[2021-09-09T09:46:40,111][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-identity-provider]
[2021-09-09T09:46:40,111][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-ilm]
[2021-09-09T09:46:40,111][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-logstash]
[2021-09-09T09:46:40,111][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-ml]
[2021-09-09T09:46:40,111][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-monitoring]
[2021-09-09T09:46:40,112][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-ql]
[2021-09-09T09:46:40,112][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-rollup]
[2021-09-09T09:46:40,112][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-security]
[2021-09-09T09:46:40,112][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-sql]
[2021-09-09T09:46:40,112][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-stack]
[2021-09-09T09:46:40,112][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-voting-only-node]
[2021-09-09T09:46:40,112][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-watcher]
[2021-09-09T09:46:40,113][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] no plugins loaded
[2021-09-09T09:46:40,161][INFO ][o.e.e.NodeEnvironment    ] [node2name.dev.company.local] using [1] data paths, mounts [[/ (/dev/mapper/centos-root)]], net usable_space [43.3gb], net total_space [49.9gb], types [xfs]
[2021-09-09T09:46:40,162][INFO ][o.e.e.NodeEnvironment    ] [node2name.dev.company.local] heap size [1gb], compressed ordinary object pointers [true]
[2021-09-09T09:46:40,264][INFO ][o.e.n.Node               ] [node2name.dev.company.local] node name [node2name.dev.company.local], node ID [z0BPC1qZS0218vDUDIFpHg], cluster name [elkdev]
[2021-09-09T09:46:43,965][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node2name.dev.company.local] [controller/206668] [Main.cc@114] controller (64 bit): Version 7.9.0 (Build 2639177a4c3ad6) Copyright (c) 2020 Elasticsearch BV
[2021-09-09T09:46:44,637][INFO ][o.e.x.s.a.s.FileRolesStore] [node2name.dev.company.local] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2021-09-09T09:46:45,849][INFO ][o.e.d.DiscoveryModule    ] [node2name.dev.company.local] using discovery type [zen] and seed hosts providers [settings]
[2021-09-09T09:46:46,299][WARN ][o.e.g.DanglingIndicesState] [node2name.dev.company.local] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-09-09T09:46:46,821][INFO ][o.e.n.Node               ] [node2name.dev.company.local] initialized
[2021-09-09T09:46:46,822][INFO ][o.e.n.Node               ] [node2name.dev.company.local] starting ...
[2021-09-09T09:46:46,973][INFO ][o.e.t.TransportService   ] [node2name.dev.company.local] publish_address {10.xxx.xxx.xx2:9300}, bound_addresses {[::]:9300}
[2021-09-09T09:46:47,147][INFO ][o.e.b.BootstrapChecks    ] [node2name.dev.company.local] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2021-09-09T09:46:57,165][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node1name.dev.company.local, node2name.dev.company.local, node3name.dev.company.local] to bootstrap a cluster: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-09T09:47:07,169][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node1name.dev.company.local, node2name.dev.company.local, node3name.dev.company.local] to bootstrap a cluster: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-09T09:47:17,166][WARN ][o.e.n.Node               ] [node2name.dev.company.local] timed out while waiting for initial discovery state - timeout: 30s
[2021-09-09T09:47:17,173][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node1name.dev.company.local, node2name.dev.company.local, node3name.dev.company.local] to bootstrap a cluster: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-09T09:47:17,184][INFO ][o.e.h.AbstractHttpServerTransport] [node2name.dev.company.local] publish_address {10.xxx.xxx.xx2:9200}, bound_addresses {[::]:9200}
[2021-09-09T09:47:17,185][INFO ][o.e.n.Node               ] [node2name.dev.company.local] started
[2021-09-09T09:47:27,176][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node1name.dev.company.local, node2name.dev.company.local, node3name.dev.company.local] to bootstrap a cluster: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-09T09:47:37,179][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node1name.dev.company.local, node2name.dev.company.local, node3name.dev.company.local] to bootstrap a cluster: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-09T09:47:47,183][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node1name.dev.company.local, node2name.dev.company.local, node3name.dev.company.local] to bootstrap a cluster: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-09T09:47:50,691][WARN ][r.suppressed             ] [node2name.dev.company.local] path: /_license, params: {}

I just have no idea where to go from here.

Okay, this was a firewall issue...port 9300 wasn't open (though I thought I had opened it).