Kibana server is not ready yet (license: null)

jceddy · September 7, 2021, 3:02pm

I am attempting to set up an Elasticsearch cluster consisting of three master-eligible nodes.

elasticsearch.yml looks like this on all of the nodes:

cluster.name: elkdev
node.name: {machinename}.dev.company.local
network.host: 0.0.0.0
http.port: 9200
node.master: false
node.data: true
node.ingest: false
discovery.seed_hosts: ["{node1name}.dev.company.local", "{node2name}.dev.company.local", "{node3name}.dev.company.local"]
cluster.initial_master_nodes: ["{node1name}.dev.company.local"]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

So, they all have the fully qualified host name as node.name, all of the fully qualified host names are included in discovery.seed_hosts, and the fully qualified host name for the FIRST node is contained in cluster.initial_master_nodes for all of the nodes.

If I run curl http://{node1name}.dev.company.local:9200/_xpack?pretty, it looks like it is started up and running successfully. Maybe importantly, it contains license information:

  "license" : {
    "uid" : "8dd0486a-9f55-4c92-8711-2c32336a746c",
    "type" : "basic",
    "mode" : "basic",
    "status" : "active"
  },

If I run the same command for either of the other nodes, I get "license": null in the output.

On nodes 2 & 3, I see this error in the log:

[2021-09-07T09:54:08,288][WARN ][r.suppressed             ] [{machinename}.dev.company.local] path: /logstash, params: {index=logstash}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) [elasticsearch-7.9.0.jar:7.9.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]

If I run journalctl -u kibana.service, I see this kind of thing in the output:

Sep 07 09:58:45 {machinename}.dev.company.local kibana[121433]: {"type":"log","@timestamp":"2021-09-07T14:58:45Z","tags":["warning","savedobjects-service"],"pid":121433,"message":"Unable to connect to Elasticsearch. Error: Request Timeout after 30000ms"}

I've been searching this forum and elsewhere for similar posts for a few days, and so far have not found a suggestion that works. Anyone have any ideas for further debugging?

Cheers,
Joe

warkolm · September 7, 2021, 10:31pm

That's the reason. Is there more to the logs from that node?

jceddy · September 8, 2021, 2:26pm

This is from the same node (node 2).

In here, 10.xxx.xxx.xx1 is the IP of node 1 (the one that seems to be working), 10.xxx.xxx.xx2 is the IP of node 2 (the node that the log is from), and 10.xxx.xxx.xx3 is the IP of node 3 (the one that is behaving the same as node 2).

[2021-09-08T09:09:26,580][WARN ][r.suppressed             ] [node2name.dev.company.locall] path: /_license, params: {}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) [elasticsearch-7.9.0.jar:7.9.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-08T09:09:26,652][WARN ][r.suppressed             ] [node2name.dev.company.local] path: /_template/logstash, params: {name=logstash}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) [elasticsearch-7.9.0.jar:7.9.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-08T09:09:29,505][WARN ][r.suppressed             ] [node2name.dev.company.local] path: /.kibana_task_manager, params: {index=.kibana_task_manager}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) [elasticsearch-7.9.0.jar:7.9.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-08T09:09:30,925][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{DxzxkKlNRFeH4yCZA_ZYbw}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dlrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-08T09:09:40,926][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{DxzxkKlNRFeH4yCZA_ZYbw}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dlrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-08T09:09:50,328][WARN ][r.suppressed             ] [node2name.dev.company.local] path: /.kibana, params: {index=.kibana}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) [elasticsearch-7.9.0.jar:7.9.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-08T09:09:50,927][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{DxzxkKlNRFeH4yCZA_ZYbw}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dlrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-08T09:09:56,628][WARN ][r.suppressed             ] [node2name.dev.company.local] path: /_license, params: {}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) [elasticsearch-7.9.0.jar:7.9.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-08T09:09:56,631][WARN ][r.suppressed             ] [node2name.dev.company.local] path: /_template/logstash, params: {name=logstash}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) [elasticsearch-7.9.0.jar:7.9.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-08T09:09:56,684][WARN ][r.suppressed             ] [node2name.dev.company.local] path: /logstash, params: {index=logstash}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) [elasticsearch-7.9.0.jar:7.9.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-08T09:10:00,928][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{DxzxkKlNRFeH4yCZA_ZYbw}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dlrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-08T09:10:02,008][WARN ][r.suppressed             ] [node2name.dev.company.local] path: /.kibana_task_manager, params: {index=.kibana_task_manager}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) [elasticsearch-7.9.0.jar:7.9.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-08T09:10:10,928][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{DxzxkKlNRFeH4yCZA_ZYbw}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dlrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0

jceddy · September 8, 2021, 3:21pm

And here is the beginning of the log up to when the errors start happening:

[2021-09-08T10:17:56,905][INFO ][o.e.n.Node               ] [node2name.dev.company.local] version[7.9.0], pid[200627], build[default/rpm/a479a2a7fce0389512d6a9361301708b92dff667/2020-08-11T21:36:48.204330Z], OS[Linux/3.10.0-1160.36.2.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/14.0.1/14.0.1+7]
[2021-09-08T10:17:56,909][INFO ][o.e.n.Node               ] [node2name.dev.company.local] JVM home [/usr/share/elasticsearch/jdk]
[2021-09-08T10:17:56,909][INFO ][o.e.n.Node               ] [node2name.dev.company.local] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-10713857089481507232, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2021-09-08T10:17:59,091][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [aggs-matrix-stats]
[2021-09-08T10:17:59,091][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [analysis-common]
[2021-09-08T10:17:59,092][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [constant-keyword]
[2021-09-08T10:17:59,092][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [flattened]
[2021-09-08T10:17:59,092][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [frozen-indices]
[2021-09-08T10:17:59,092][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [ingest-common]
[2021-09-08T10:17:59,093][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [ingest-geoip]
[2021-09-08T10:17:59,093][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [ingest-user-agent]
[2021-09-08T10:17:59,093][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [kibana]
[2021-09-08T10:17:59,093][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [lang-expression]
[2021-09-08T10:17:59,094][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [lang-mustache]
[2021-09-08T10:17:59,094][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [lang-painless]
[2021-09-08T10:17:59,094][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [mapper-extras]
[2021-09-08T10:17:59,094][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [parent-join]
[2021-09-08T10:17:59,094][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [percolator]
[2021-09-08T10:17:59,095][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [rank-eval]
[2021-09-08T10:17:59,095][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [reindex]
[2021-09-08T10:17:59,095][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [repository-url]
[2021-09-08T10:17:59,095][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [search-business-rules]
[2021-09-08T10:17:59,095][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [searchable-snapshots]
[2021-09-08T10:17:59,095][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [spatial]
[2021-09-08T10:17:59,096][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [systemd]
[2021-09-08T10:17:59,096][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [tasks]
[2021-09-08T10:17:59,096][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [transform]
[2021-09-08T10:17:59,096][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [transport-netty4]
[2021-09-08T10:17:59,096][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [vectors]
[2021-09-08T10:17:59,096][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [wildcard]
[2021-09-08T10:17:59,097][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-analytics]
[2021-09-08T10:17:59,097][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-async]
[2021-09-08T10:17:59,097][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-async-search]
[2021-09-08T10:17:59,097][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-autoscaling]
[2021-09-08T10:17:59,098][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-ccr]
[2021-09-08T10:17:59,098][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-core]
[2021-09-08T10:17:59,098][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-data-streams]
[2021-09-08T10:17:59,098][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-deprecation]
[2021-09-08T10:17:59,098][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-enrich]
[2021-09-08T10:17:59,098][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-eql]
[2021-09-08T10:17:59,099][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-graph]
[2021-09-08T10:17:59,099][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-identity-provider]
[2021-09-08T10:17:59,099][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-ilm]
[2021-09-08T10:17:59,099][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-logstash]
[2021-09-08T10:17:59,099][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-ml]
[2021-09-08T10:17:59,100][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-monitoring]
[2021-09-08T10:17:59,100][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-ql]
[2021-09-08T10:17:59,100][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-rollup]
[2021-09-08T10:17:59,100][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-security]
[2021-09-08T10:17:59,100][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-sql]
[2021-09-08T10:17:59,101][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-stack]
[2021-09-08T10:17:59,101][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-voting-only-node]
[2021-09-08T10:17:59,101][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-watcher]
[2021-09-08T10:17:59,102][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] no plugins loaded
[2021-09-08T10:17:59,154][INFO ][o.e.e.NodeEnvironment    ] [node2name.dev.company.local] using [1] data paths, mounts [[/ (/dev/mapper/centos-root)]], net usable_space [43.3gb], net total_space [49.9gb], types [xfs]
[2021-09-08T10:17:59,154][INFO ][o.e.e.NodeEnvironment    ] [node2name.dev.company.local] heap size [1gb], compressed ordinary object pointers [true]
[2021-09-08T10:17:59,230][INFO ][o.e.n.Node               ] [node2name.dev.company.local] node name [node2name.dev.company.local], node ID [z0BPC1qZS0218vDUDIFpHg], cluster name [elkdev]
[2021-09-08T10:18:02,695][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node2name.dev.company.local] [controller/200855] [Main.cc@114] controller (64 bit): Version 7.9.0 (Build 2639177a4c3ad6) Copyright (c) 2020 Elasticsearch BV
[2021-09-08T10:18:03,530][INFO ][o.e.x.s.a.s.FileRolesStore] [node2name.dev.company.local] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2021-09-08T10:18:04,627][INFO ][o.e.d.DiscoveryModule    ] [node2name.dev.company.local] using discovery type [zen] and seed hosts providers [settings]
[2021-09-08T10:18:05,085][WARN ][o.e.g.DanglingIndicesState] [node2name.dev.company.local] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-09-08T10:18:05,485][INFO ][o.e.n.Node               ] [node2name.dev.company.local] initialized
[2021-09-08T10:18:05,486][INFO ][o.e.n.Node               ] [node2name.dev.company.local] starting ...
[2021-09-08T10:18:05,649][INFO ][o.e.t.TransportService   ] [node2name.dev.company.local] publish_address {10.xxx.xxx.xx2:9300}, bound_addresses {[::]:9300}
[2021-09-08T10:18:05,813][INFO ][o.e.b.BootstrapChecks    ] [node2name.dev.company.local] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2021-09-08T10:18:15,827][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{BUP6LidrQcypP_WjQyAkZQ}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dlrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-08T10:18:25,830][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{BUP6LidrQcypP_WjQyAkZQ}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dlrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-08T10:18:35,831][WARN ][o.e.n.Node               ] [node2name.dev.company.local] timed out while waiting for initial discovery state - timeout: 30s
[2021-09-08T10:18:35,833][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{BUP6LidrQcypP_WjQyAkZQ}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dlrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-08T10:18:35,848][INFO ][o.e.h.AbstractHttpServerTransport] [node2name.dev.company.local] publish_address {10.xxx.xxx.xx2:9200}, bound_addresses {[::]:9200}
[2021-09-08T10:18:35,849][INFO ][o.e.n.Node               ] [node2name.dev.company.local] started

jceddy · September 8, 2021, 5:14pm

Oh, correction to my original post:

The first node is configured with this:

node.master: true
node.data: false

The other two nodes are configured as mentioned in the original post.

There aren't all currently set up with node.master: true because I thought that might be causing an issue.

Should they all have both node.master: true and node.data: true if I want them all to be master-eligible data nodes?

jceddy · September 8, 2021, 6:29pm

I tried a couple of things, including setting

node.master: true
node.data: true
node.ingest: true

On all of the nodes, but still no luck.

jceddy · September 9, 2021, 2:53pm

I made a change to the configuration to add all of the nodes to the initial master nodes list:

cluster.name: elkdev
node.name: node2name.dev.company.local
network.host: 0.0.0.0
http.port: 9200
node.master: true
node.data: true
node.ingest: true
discovery.seed_hosts: ["node1name.dev.company.local", "node2name.dev.company.local", "node3name.dev.company.local"]
cluster.initial_master_nodes: ["node1name.dev.company.local", "node2name.dev.company.local", "node3name.dev.company.local"]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

Still getting the null exceptions, here is the log from node 2 leading up to the first exception:

[2021-09-09T09:46:40,108][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [transform]
[2021-09-09T09:46:40,109][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [transport-netty4]
[2021-09-09T09:46:40,109][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [vectors]
[2021-09-09T09:46:40,109][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [wildcard]
[2021-09-09T09:46:40,109][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-analytics]
[2021-09-09T09:46:40,109][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-async]
[2021-09-09T09:46:40,109][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-async-search]
[2021-09-09T09:46:40,110][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-autoscaling]
[2021-09-09T09:46:40,110][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-ccr]
[2021-09-09T09:46:40,110][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-core]
[2021-09-09T09:46:40,110][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-data-streams]
[2021-09-09T09:46:40,110][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-deprecation]
[2021-09-09T09:46:40,110][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-enrich]
[2021-09-09T09:46:40,110][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-eql]
[2021-09-09T09:46:40,111][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-graph]
[2021-09-09T09:46:40,111][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-identity-provider]
[2021-09-09T09:46:40,111][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-ilm]
[2021-09-09T09:46:40,111][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-logstash]
[2021-09-09T09:46:40,111][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-ml]
[2021-09-09T09:46:40,111][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-monitoring]
[2021-09-09T09:46:40,112][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-ql]
[2021-09-09T09:46:40,112][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-rollup]
[2021-09-09T09:46:40,112][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-security]
[2021-09-09T09:46:40,112][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-sql]
[2021-09-09T09:46:40,112][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-stack]
[2021-09-09T09:46:40,112][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-voting-only-node]
[2021-09-09T09:46:40,112][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] loaded module [x-pack-watcher]
[2021-09-09T09:46:40,113][INFO ][o.e.p.PluginsService     ] [node2name.dev.company.local] no plugins loaded
[2021-09-09T09:46:40,161][INFO ][o.e.e.NodeEnvironment    ] [node2name.dev.company.local] using [1] data paths, mounts [[/ (/dev/mapper/centos-root)]], net usable_space [43.3gb], net total_space [49.9gb], types [xfs]
[2021-09-09T09:46:40,162][INFO ][o.e.e.NodeEnvironment    ] [node2name.dev.company.local] heap size [1gb], compressed ordinary object pointers [true]
[2021-09-09T09:46:40,264][INFO ][o.e.n.Node               ] [node2name.dev.company.local] node name [node2name.dev.company.local], node ID [z0BPC1qZS0218vDUDIFpHg], cluster name [elkdev]
[2021-09-09T09:46:43,965][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node2name.dev.company.local] [controller/206668] [Main.cc@114] controller (64 bit): Version 7.9.0 (Build 2639177a4c3ad6) Copyright (c) 2020 Elasticsearch BV
[2021-09-09T09:46:44,637][INFO ][o.e.x.s.a.s.FileRolesStore] [node2name.dev.company.local] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2021-09-09T09:46:45,849][INFO ][o.e.d.DiscoveryModule    ] [node2name.dev.company.local] using discovery type [zen] and seed hosts providers [settings]
[2021-09-09T09:46:46,299][WARN ][o.e.g.DanglingIndicesState] [node2name.dev.company.local] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-09-09T09:46:46,821][INFO ][o.e.n.Node               ] [node2name.dev.company.local] initialized
[2021-09-09T09:46:46,822][INFO ][o.e.n.Node               ] [node2name.dev.company.local] starting ...
[2021-09-09T09:46:46,973][INFO ][o.e.t.TransportService   ] [node2name.dev.company.local] publish_address {10.xxx.xxx.xx2:9300}, bound_addresses {[::]:9300}
[2021-09-09T09:46:47,147][INFO ][o.e.b.BootstrapChecks    ] [node2name.dev.company.local] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2021-09-09T09:46:57,165][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node1name.dev.company.local, node2name.dev.company.local, node3name.dev.company.local] to bootstrap a cluster: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-09T09:47:07,169][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node1name.dev.company.local, node2name.dev.company.local, node3name.dev.company.local] to bootstrap a cluster: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-09T09:47:17,166][WARN ][o.e.n.Node               ] [node2name.dev.company.local] timed out while waiting for initial discovery state - timeout: 30s
[2021-09-09T09:47:17,173][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node1name.dev.company.local, node2name.dev.company.local, node3name.dev.company.local] to bootstrap a cluster: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-09T09:47:17,184][INFO ][o.e.h.AbstractHttpServerTransport] [node2name.dev.company.local] publish_address {10.xxx.xxx.xx2:9200}, bound_addresses {[::]:9200}
[2021-09-09T09:47:17,185][INFO ][o.e.n.Node               ] [node2name.dev.company.local] started
[2021-09-09T09:47:27,176][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node1name.dev.company.local, node2name.dev.company.local, node3name.dev.company.local] to bootstrap a cluster: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-09T09:47:37,179][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node1name.dev.company.local, node2name.dev.company.local, node3name.dev.company.local] to bootstrap a cluster: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-09T09:47:47,183][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2name.dev.company.local] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node1name.dev.company.local, node2name.dev.company.local, node3name.dev.company.local] to bootstrap a cluster: have discovered [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [10.xxx.xxx.xx1:9300, 10.xxx.xxx.xx3:9300] from hosts providers and [{node2name.dev.company.local}{z0BPC1qZS0218vDUDIFpHg}{NqRlnRAaSxSoHkWdRvdGHg}{10.xxx.xxx.xx2}{10.xxx.xxx.xx2:9300}{dilmrt}{ml.machine_memory=67020705792, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-09T09:47:50,691][WARN ][r.suppressed             ] [node2name.dev.company.local] path: /_license, params: {}

I just have no idea where to go from here.

jceddy · September 10, 2021, 3:39pm

Okay, this was a firewall issue...port 9300 wasn't open (though I thought I had opened it).

system · October 8, 2021, 3:40pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.