# Nodes are offline **URL:** \ **Category:** Elasticsearch\ **Tags:** docker\ **Created:** [June 11, 2023, 6:41am UTC](https://discuss.elastic.co/t/nodes-are-offline/335716 "2023-06-11T06:41:42Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Siavash\_Fazli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/siavash_fazli/32/103915_2.png) [@Siavash\_Fazli](https://discuss.elastic.co/u/Siavash_Fazli)\ **Post date:** [June 11, 2023, 6:41am UTC](https://discuss.elastic.co/t/nodes-are-offline/335716/1 "2023-06-11T06:41:42Z")
Hi, dears. I deployed a 3-node elastic cluster with default docker-compose in the Elastic document. but expect the master node, other nodes are offline. Are there any additional settings that I have to do? also why master node's role is N/A? ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/f/0f1387612ccd79942b8c98c611583ee985a91819.png) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 11, 2023, 6:46am UTC](https://discuss.elastic.co/t/nodes-are-offline/335716/2 "2023-06-11T06:46:31Z")
Which version of Elasticsearch are you using? What does your node configurations look like? What is in the Elasticsearch logs? ---
**Author:** ![Siavash\_Fazli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/siavash_fazli/32/103915_2.png) [@Siavash\_Fazli](https://discuss.elastic.co/u/Siavash_Fazli)\ **Post date:** [June 11, 2023, 7:02am UTC](https://discuss.elastic.co/t/nodes-are-offline/335716/3 "2023-06-11T07:02:21Z")
I'm using version 8.8.0. elasticsearch.yml for master node: ```auto network.host: 0.0.0.0 node.name: es01-master node.roles: ["master", "remote_cluster_client"] discovery.type: multi-node cluster.initial_master_nodes: es01-master discovery.seed_hosts: ["es02-data", "es03-data"] bootstrap.memory_lock: true xpack.security.enabled: true ingest.geoip.downloader.enabled: false xpack.monitoring.collection.enabled: true ``` Also, I have two data nodes that their elasticsearch.yml's are same: ```auto network.host: 0.0.0.0 node.name: es02-data node.roles: ["data"] discovery.type: multi-node cluster.initial_master_nodes: es01-master discovery.seed_hosts: ["es01-master", "es03-data"] bootstrap.memory_lock: true xpack.security.enabled: true ingest.geoip.downloader.enabled: false ``` There is not a unusual logs in log file, but when I grep data nodes in master node container I got this log: ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/4/c49a6fed5add43004859e66cd4d5b1ab2ef72788.png) @Christian_Dahlqvist ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 11, 2023, 7:06am UTC](https://discuss.elastic.co/t/nodes-are-offline/335716/4 "2023-06-11T07:06:37Z")
> [@Siavash\_Fazli](#): > > `discovery.seed_hosts: ["es01-master", "es03-data"]` Does this resolve to the correct IP address? ---
**Author:** ![Siavash\_Fazli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/siavash_fazli/32/103915_2.png) [@Siavash\_Fazli](https://discuss.elastic.co/u/Siavash_Fazli)\ **Post date:** [June 11, 2023, 7:16am UTC](https://discuss.elastic.co/t/nodes-are-offline/335716/5 "2023-06-11T07:16:28Z")
they are connected in two networks. I use `curl` for connection checking and there are seeing each other. ```auto root@elk1:/opt/elk# docker inspect es01-master | grep IPAddress "SecondaryIPAddresses": null, "IPAddress": "", "IPAddress": "192.168.176.3", "IPAddress": "172.22.0.3", root@elk1:/opt/elk# docker inspect es02-data | grep IPAddress "SecondaryIPAddresses": null, "IPAddress": "", "IPAddress": "192.168.176.4", "IPAddress": "172.22.0.4", root@elk1:/opt/elk# docker inspect es03-data | grep IPAddress "SecondaryIPAddresses": null, "IPAddress": "", "IPAddress": "192.168.176.5", "IPAddress": "172.22.0.5", ``` ---
**Author:** ![Siavash\_Fazli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/siavash_fazli/32/103915_2.png) [@Siavash\_Fazli](https://discuss.elastic.co/u/Siavash_Fazli)\ **Post date:** [June 11, 2023, 10:34am UTC](https://discuss.elastic.co/t/nodes-are-offline/335716/6 "2023-06-11T10:34:23Z")
@Christian_Dahlqvist ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 11, 2023, 12:20pm UTC](https://discuss.elastic.co/t/nodes-are-offline/335716/7 "2023-06-11T12:20:35Z")
Your `discovery.seed_hosts` is wrong, this setting needs to have **only** the master eligibles nodes, your `es03-data` is not master eligible, you need to remove it from this setting, I don't think this is the issue, but you need to remove it. Since you have only `es01-master` as master-eligible, only this node should be listed in the `discovery.seed_hosts` setting. Are you sure this isn't an issue with the monitoring? The self-monitoring you are using is deprecated, should not be used anymore and may have some issues. Do you have any **WARN** or **ERROR** logs in your master log? The logs you shared indicate no issue. What is the result of a [`_cluster/health`](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html) request to you master node? What do you have in the log of the other nodes? ---
**Author:** ![Siavash\_Fazli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/siavash_fazli/32/103915_2.png) [@Siavash\_Fazli](https://discuss.elastic.co/u/Siavash_Fazli)\ **Post date:** [June 14, 2023, 8:54am UTC](https://discuss.elastic.co/t/nodes-are-offline/335716/8 "2023-06-14T08:54:44Z")
I change these variables in three elasticsearch.yml: ```auto cluster.initial_master_nodes: es01-master discovery.seed_hosts: es01-master ``` but nothing changed. I didn't configure metricbeat yet. And there isn't any **WARN** or **ERROR** in logs. stdout of cluster/health : ```auto { "cluster_name": "naringames_elastic_cluster", "status": "green", "timed_out": false, "number_of_nodes": 3, "number_of_data_nodes": 2, "active_primary_shards": 375, "active_shards": 750, "relocating_shards": 0, "initializing_shards": 0, "unassigned_shards": 0, "delayed_unassigned_shards": 0, "number_of_pending_tasks": 0, "number_of_in_flight_fetch": 0, "task_max_waiting_in_queue_millis": 0, "active_shards_percent_as_number": 100 } ``` other node log: ```auto es02-data | {"@timestamp":"2023-06-14T00:28:39.819Z", "log.level": "WARN", "message":"this node is locked into cluster UUID [irImass0TMyQS-ofDmTgKw] but [cluster.initial_master_nodes] is set to [es01-master]; remove this setting to avoid possible data loss caused by subsequent cluster bootstrap attempts; for further information see https://www.elastic.co/guide/en/elasticsearch/reference/8.8/important-settings.html#initial_master_nodes", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02-data][scheduler][T#1]","log.logger":"org.elasticsearch.cluster.coordination.ClusterBootstrapService","elasticsearch.cluster.uuid":"irImass0TMyQS-ofDmTgKw","elasticsearch.node.id":"PvNPsykNTuCPFnkdOHF9Ig","elasticsearch.node.name":"es02-data","elasticsearch.cluster.name":"naringames_elastic_cluster"} es02-data | {"@timestamp":"2023-06-14T07:54:02.941Z", "log.level": "INFO", "message":"master node [{es01-master}{4OGHbn5iQH-tWnUe6KT6UA}{DYOu_W-CSM-Q8letxLLBlw}{es01-master}{172.22.0.3}{172.22.0.3:9300}{mr}{8.8.0}] disconnected, restarting discovery", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02-data][cluster_coordination][T#1]","log.logger":"org.elasticsearch.cluster.coordination.Coordinator","elasticsearch.cluster.uuid":"irImass0TMyQS-ofDmTgKw","elasticsearch.node.id":"PvNPsykNTuCPFnkdOHF9Ig","elasticsearch.node.name":"es02-data","elasticsearch.cluster.name":"naringames_elastic_cluster"} es02-data | {"@timestamp":"2023-06-14T07:55:15.981Z", "log.level": "WARN", "message":"this node is locked into cluster UUID [irImass0TMyQS-ofDmTgKw] but [cluster.initial_master_nodes] is set to [es01-master]; remove this setting to avoid possible data loss caused by subsequent cluster bootstrap attempts; for further information see https://www.elastic.co/guide/en/elasticsearch/reference/8.8/important-settings.html#initial_master_nodes", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.cluster.coordination.ClusterBootstrapService","elasticsearch.node.name":"es02-data","elasticsearch.cluster.name":"naringames_elastic_cluster"} es02-data | {"@timestamp":"2023-06-14T07:55:19.922Z", "log.level": "INFO", "message":"master node changed {previous [], current [{es01-master}{4OGHbn5iQH-tWnUe6KT6UA}{9UTrBDTjQMCCk9QEA8Ef8w}{es01-master}{172.22.0.3}{172.22.0.3:9300}{mr}{8.8.0}]}, added {{es01-master}{4OGHbn5iQH-tWnUe6KT6UA}{9UTrBDTjQMCCk9QEA8Ef8w}{es01-master}{172.22.0.3}{172.22.0.3:9300}{mr}{8.8.0}, {es03-data}{UpA2j6uARM2h10-6UVrCRg}{B-31WnipR1SWFt8xXypOSQ}{es03-data}{172.22.0.5}{172.22.0.5:9300}{d}{8.8.0}}, term: 3, version: 2077, reason: ApplyCommitRequest{term=3, version=2077, sourceNode={es01-master}{4OGHbn5iQH-tWnUe6KT6UA}{9UTrBDTjQMCCk9QEA8Ef8w}{es01-master}{172.22.0.3}{172.22.0.3:9300}{mr}{8.8.0}{xpack.installed=true}}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02-data][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.ClusterApplierService","elasticsearch.node.name":"es02-data","elasticsearch.cluster.name":"naringames_elastic_cluster"} ``` @leandrojmp ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 14, 2023, 12:54pm UTC](https://discuss.elastic.co/t/nodes-are-offline/335716/9 "2023-06-14T12:54:38Z")
> [@Siavash\_Fazli](#): > > ```auto > "status": "green", > "timed_out": false, > "number_of_nodes": 3, > "number_of_data_nodes": 2 > > ``` Your cluster is _green_, this means that it is working without any issue, it also shows that you have 3 nodes in the cluster and 2 of them are data nodes, this match what you shared in your docker compose. I see no issue with offline cluster, as I said this looks like an issue with monitoring. Self-monitoring is deprecated and not recommended anymore, you should use metricbeat to monitor your cluster, but if I'm not wrong for self-monitoring to work you need this setting in every node: `xpack.monitoring.collection.enabled: true`, from what you shared you only have this for the master node. > [@Siavash\_Fazli](#): > > `es02-data | {"@timestamp":"2023-06-14T07:55:15.981Z", "log.level": "WARN", "message":"this node is locked into cluster UUID [irImass0TMyQS-ofDmTgKw] but [cluster.initial_master_nodes] is set to [es01-master]; remove this setting to avoid possible data loss caused by subsequent cluster bootstrap attempts; for further information see https://www.elastic.co/guide/en/elasticsearch/reference/8.8/important-settings.html#initial_master_nodes", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.cluster.coordination.ClusterBootstrapService","elasticsearch.node.name":"es02-data","elasticsearch.cluster.name":"naringames_elastic_cluster"}` This is a warning that happens on a restart, it means that you still have the setting `cluster.initial_master_nodes` in your `elasticsearch.yml`, this setting needs to be removed after the cluster is formed for the first time, remove it and the warn will disappear next time you restart your cluster. ---
**Author:** ![Siavash\_Fazli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/siavash_fazli/32/103915_2.png) [@Siavash\_Fazli](https://discuss.elastic.co/u/Siavash_Fazli)\ **Post date:** [June 24, 2023, 8:25am UTC](https://discuss.elastic.co/t/nodes-are-offline/335716/10 "2023-06-24T08:25:37Z")
@leandrojmp "I'm grateful that I was finally able to enable online monitoring using `xpack.monitoring.collection.enabled: true`. I have another question. Could you please explain how the nodes function? For instance, if I run multiple pipelines with logstash, will the cluster distribute the workload across the nodes to alleviate pressure on the central server and prevent it from reaching its CPU capacity? Additionally, do you have any documentation available that covers the management of requests and queries within a cluster?" ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 22, 2023, 8:26am UTC](https://discuss.elastic.co/t/nodes-are-offline/335716/11 "2023-07-22T08:26:15Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.