# Add elasticsearch node multiple machine **URL:** \ **Category:** Elasticsearch\ **Created:** [March 13, 2020, 4:07am UTC](https://discuss.elastic.co/t/add-elasticsearch-node-multiple-machine/223443 "2020-03-13T04:07:02Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![Charles\_Yuliansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charles_yuliansen/32/45664_2.png) [@Charles\_Yuliansen](https://discuss.elastic.co/u/Charles_Yuliansen)\ **Post date:** [March 13, 2020, 4:07am UTC](https://discuss.elastic.co/t/add-elasticsearch-node-multiple-machine/223443/1 "2020-03-13T04:07:02Z")
Hi, im currently doing some poc about add node to elasticsearch cluster with multiple machine. The master node should be x.x.x.246 and the data node that i want to add is x.x.x.99. Running the latest elasticsearch 7.6. I've make sure that both elastic are on the same version. # Here is my config file for master node. ```auto cluster.name : elasticsearch node.name : Node_master node.master : true node.data : false network.host : 10.64.2.246 http.port : 9200 transport.tcp.port: 9300 discovery.seed_hosts: - x.x.x.246:9300 - x.x.x.99:9301 cluster.initial_master_nodes: - x.x.x.246:9300 - x.x.x.99:9301 ``` # here is my config for data node ```auto cluster.name : elasticsearch node.name : Node-1 node.master : false node.data : true network.host : 10.64.2.99 http.port : 9201 transport.tcp.port: 9301 discovery.seed_hosts: - x.x.x.246:9300 - x.x.x.99:9301 cluster.initial_master_nodes: - x.x.x.246:9300 - x.x.x.99:9301 ``` ## here is the error returned when i restart elasticsearch on master node : ```auto [root@localhost elasticsearch]# service elasticsearch start Starting elasticsearch (via systemctl): Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details. [FAILED] ``` systemctl view on master node: ```auto [root@localhost elasticsearch]# systemctl status elasticsearch.service -l ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Fri 2020-03-13 11:37:44 WIB; 2min 20s ago Docs: http://www.elastic.co Process: 127072 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE) Main PID: 127072 (code=exited, status=1/FAILURE) Mar 13 11:37:43 localhost.localdomain elasticsearch[127072]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) Mar 13 11:37:43 localhost.localdomain elasticsearch[127072]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) Mar 13 11:37:43 localhost.localdomain elasticsearch[127072]: at org.elasticsearch.cli.Command.main(Command.java:90) Mar 13 11:37:43 localhost.localdomain elasticsearch[127072]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) Mar 13 11:37:43 localhost.localdomain elasticsearch[127072]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) Mar 13 11:37:43 localhost.localdomain elasticsearch[127072]: For complete error details, refer to the log at /var/log/elasticsearch/elasticsearch.log Mar 13 11:37:44 localhost.localdomain systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE Mar 13 11:37:44 localhost.localdomain systemd[1]: Failed to start Elasticsearch. Mar 13 11:37:44 localhost.localdomain systemd[1]: Unit elasticsearch.service entered failed state. Mar 13 11:37:44 localhost.localdomain systemd[1]: elasticsearch.service failed. ``` ## here is the error returned when i restart elasticsearch on data node : ```auto [root@dev elasticsearch]# service elasticsearch restart Restarting elasticsearch (via systemctl): Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details. [FAILED] ``` ```auto [root@dev elasticsearch]# systemctl status elasticsearch ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Fri 2020-03-13 10:29:21 WIB; 16s ago Docs: http://www.elastic.co Process: 10474 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE) Main PID: 10474 (code=exited, status=1/FAILURE) ``` systemctl view on data node: ```auto ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Fri 2020-03-13 10:29:21 WIB; 15min ago Docs: http://www.elastic.co Process: 10474 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE) Main PID: 10474 (code=exited, status=1/FAILURE) Mar 13 10:29:21 dev.int.cbn.net.id elasticsearch[10474]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) Mar 13 10:29:21 dev.int.cbn.net.id elasticsearch[10474]: at org.elasticsearch.cli.Command.main(Command.java:90) Mar 13 10:29:21 dev.int.cbn.net.id elasticsearch[10474]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) Mar 13 10:29:21 dev.int.cbn.net.id elasticsearch[10474]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) Mar 13 10:29:21 dev.int.cbn.net.id elasticsearch[10474]: Caused by: com.fasterxml.jackson.dataformat.yaml.snakeyaml.error.MarkedYAMLException: while scanning a simple key Mar 13 10:29:21 dev.int.cbn.net.id elasticsearch[10474]: in 'reader', line 25, column 1: Mar 13 10:29:21 dev.int.cbn.net.id systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE Mar 13 10:29:21 dev.int.cbn.net.id systemd[1]: Failed to start Elasticsearch. Mar 13 10:29:21 dev.int.cbn.net.id systemd[1]: Unit elasticsearch.service entered failed state. Mar 13 10:29:21 dev.int.cbn.net.id systemd[1]: elasticsearch.service failed. ``` any answer or explanation will be appreciated so much. Thank you ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 13, 2020, 4:11am UTC](https://discuss.elastic.co/t/add-elasticsearch-node-multiple-machine/223443/2 "2020-03-13T04:11:56Z")
> [@Charles\_Yuliansen](#): > > refer to the log at /var/log/elasticsearch/elasticsearch.log Can you check that and post the output? ---
**Author:** ![Charles\_Yuliansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charles_yuliansen/32/45664_2.png) [@Charles\_Yuliansen](https://discuss.elastic.co/u/Charles_Yuliansen)\ **Post date:** [March 13, 2020, 4:21am UTC](https://discuss.elastic.co/t/add-elasticsearch-node-multiple-machine/223443/3 "2020-03-13T04:21:28Z")
hi there, thank you for your respond. here is the log at master node x.x.x.246: ```auto [root@localhost elasticsearch]# tail -l elasticsearch.log -l tail: invalid option -- 'l' Try 'tail --help' for more information. [root@localhost elasticsearch]# tail -l elasticsearch.log Caused by: java.lang.IllegalStateException: Node is started with node.data=fa lse, but has shard data: [/var/lib/elasticsearch/nodes/0/indices/e3cMeIVSQdmI wVVgh_Bd0w/0, /var/lib/elasticsearch/nodes/0/indices/Prq1jnKSSoyt3XWJO6iCUg/0 , /var/lib/elasticsearch/nodes/0/indices/pnQ2AWwnSca8-hFVL_gtYA/0, /var/lib/e lasticsearch/nodes/0/indices/bFsN-CkCSjufBACUL27u0w/0, /var/lib/elasticsearch /nodes/0/indices/mvFmVLaOSN-wAvMgXknzTg/0, /var/lib/elasticsearch/nodes/0/ind ices/XBSojsDSRs6IXYQNH9j-Jg/0, /var/lib/elasticsearch/nodes/0/indices/0X9qyFy nTPac0YZvXn9Ufw/0, /var/lib/elasticsearch/nodes/0/indices/NQbHnD4CTweCU4LgfKR 77A/0, /var/lib/elasticsearch/nodes/0/indices/vDe9aUIpQ8ipqbmPEXnTFA/0, /var/ lib/elasticsearch/nodes/0/indices/97L23TQ1TJmKHDYzwaYfRg/0, /var/lib/elastics earch/nodes/0/indices/udZhKWpVSVCAsJnOVcRVLQ/0, /var/lib/elasticsearch/nodes/ 0/indices/U1b1IEMWRYqzPlDu9TcG0Q/0, /var/lib/elasticsearch/nodes/0/indices/_C LujyNjQgmCxLFBjO79Zg/0, /var/lib/elasticsearch/nodes/0/indices/LNFkjsKnQMCD5y YBHKyyNw/0, /var/lib/elasticsearch/nodes/0/indices/HnFJxvS7TveyGoawLxDFtA/0, /var/lib/elasticsearch/nodes/0/indices/pqkhAZObTj63zylELg7ySQ/0, /var/lib/ela sticsearch/nodes/0/indices/61ETZjspQzOk6q5fGn3Twg/0, /var/lib/elasticsearch/n odes/0/indices/5MC-yxWeS6WmubAOIHJt1w/0, /var/lib/elasticsearch/nodes/0/indic es/abPk_EJyTluw36cmNf1ZrQ/0, /var/lib/elasticsearch/nodes/0/indices/s-lG4szwR 8uMIQfLo_JA1w/0, /var/lib/elasticsearch/nodes/0/indices/Ii5lCjuKRfS5tW4cXEnJn g/0, /var/lib/elasticsearch/nodes/0/indices/J6S12mDCR8KGnfWScyiqpw/0, /var/li b/elasticsearch/nodes/0/indices/Sr0g5j7VTzqfcLcd1yoHDQ/0, /var/lib/elasticsea rch/nodes/0/indices/8Gk6QC7_TS2p0qRtx6XCmA/0, /var/lib/elasticsearch/nodes/0/ indices/BjMiZoZ5SIyBNbwND7fZ5w/0, /var/lib/elasticsearch/nodes/0/indices/pCmn XkAIR6a7k_Kv2uiwig/0, /var/lib/elasticsearch/nodes/0/indices/fSpz7hUCRWuX_YqT Op7FUA/0, /var/lib/elasticsearch/nodes/0/indices/uXXph5eFSPGH7RRuEKDGHQ/0, /v ar/lib/elasticsearch/nodes/0/indices/nr3MS0J9TzuElqtUjqEwqQ/0, /var/lib/elast icsearch/nodes/0/indices/SNiHj03OT2-YC8wMb7LyLg/0, /var/lib/elasticsearch/nod es/0/indices/pzmAhBOES-KtmESbs0_EKQ/0, /var/lib/elasticsearch/nodes/0/indices /cXhosVdCSayJJrf3tvvG7w/0]. Use 'elasticsearch-node repurpose' tool to clean up at org.elasticsearch.env.NodeEnvironment.ensureNoShardData(NodeEnviro nment.java:1083) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java: 326) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.node.Node.(Node.java:277) ~[elasticsearch- 7.6.1.jar:7.6.1] at org.elasticsearch.node.Node.(Node.java:257) ~[elasticsearch- 7.6.1.jar:7.6.1] at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[e lasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java: 170) ~[elasticsearch-7.6.1.jar:7.6.1] ... 6 more ``` here is the log at data node x.x.x.99: ```auto [root@dev elasticsearch]# tail -l elasticsearch.log ... 3 more [2020-03-13T09:47:27,870][INFO][o.e.c.r.a.AllocationService] [dev.int.cbn.net.id] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[g_station][0]]]). [2020-03-13T10:29:18,483][INFO][o.e.n.Node] [dev.int.cbn.net.id] stopping ... [2020-03-13T10:29:18,566][INFO][o.e.x.w.WatcherService] [dev.int.cbn.net.id] stopping watch service, reason [shutdown initiated] [2020-03-13T10:29:18,569][INFO][o.e.x.w.WatcherLifeCycleService] [dev.int.cbn.net.id] watcher has stopped and shutdown [2020-03-13T10:29:18,789][INFO][o.e.x.m.p.l.CppLogMessageHandler] [dev.int.cbn.net.id] [controller/5534] [Main.cc@150] Ml controller exiting [2020-03-13T10:29:18,795][INFO][o.e.x.m.p.NativeController] [dev.int.cbn.net.id] Native controller process has stopped - no new native processes can be started [2020-03-13T10:29:18,948][INFO][o.e.n.Node] [dev.int.cbn.net.id] stopped [2020-03-13T10:29:18,949][INFO][o.e.n.Node] [dev.int.cbn.net.id] closing ... [2020-03-13T10:29:19,004][INFO][o.e.n.Node] [dev.int.cbn.net.id] closed ``` ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 13, 2020, 4:24am UTC](https://discuss.elastic.co/t/add-elasticsearch-node-multiple-machine/223443/4 "2020-03-13T04:24:00Z")
Can you show a bit more than that please, there's not enough on either node to show what is or isn't happening. ---
**Author:** ![Charles\_Yuliansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charles_yuliansen/32/45664_2.png) [@Charles\_Yuliansen](https://discuss.elastic.co/u/Charles_Yuliansen)\ **Post date:** [March 13, 2020, 4:38am UTC](https://discuss.elastic.co/t/add-elasticsearch-node-multiple-machine/223443/5 "2020-03-13T04:38:43Z")
hi im sorry about that. here is the data node: ```auto [root@dev elasticsearch]# tail -20 elasticsearch.log at org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:180) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:104) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:95) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:106) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:45) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.lambda$asyncShardOperation$0(TransportSingleShardAction.java:110) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.1.jar:7.6.1] ... 3 more [2020-03-13T09:47:27,870][INFO][o.e.c.r.a.AllocationService] [dev.int.cbn.net.id] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[g_station][0]]]). [2020-03-13T10:29:18,483][INFO][o.e.n.Node] [dev.int.cbn.net.id] stopping ... [2020-03-13T10:29:18,566][INFO][o.e.x.w.WatcherService] [dev.int.cbn.net.id] stopping watch service, reason [shutdown initiated] [2020-03-13T10:29:18,569][INFO][o.e.x.w.WatcherLifeCycleService] [dev.int.cbn.net.id] watcher has stopped and shutdown [2020-03-13T10:29:18,789][INFO][o.e.x.m.p.l.CppLogMessageHandler] [dev.int.cbn.net.id] [controller/5534] [Main.cc@150] Ml controller exiting [2020-03-13T10:29:18,795][INFO][o.e.x.m.p.NativeController] [dev.int.cbn.net.id] Native controller process has stopped - no new native processes can be started [2020-03-13T10:29:18,948][INFO][o.e.n.Node] [dev.int.cbn.net.id] stopped [2020-03-13T10:29:18,949][INFO][o.e.n.Node] [dev.int.cbn.net.id] closing ... [2020-03-13T10:29:19,004][INFO][o.e.n.Node] [dev.int.cbn.net.id] closed ``` here is the master node : ```auto [root@localhost elasticsearch]# tail -20 elasticsearch.log at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.6.1.jar:7.6.1] [2020-03-13T12:07:51,762][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [Node_master] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Node is started with node.data=false, but has shard data: [/var/lib/elasticsearch/nodes/0/indices/e3cMeIVSQdmIwVVgh_Bd0w/0, /var/lib/elasticsearch/nodes/0/indices/Prq1jnKSSoyt3XWJO6iCUg/0, /var/lib/elasticsearch/nodes/0/indices/pnQ2AWwnSca8-hFVL_gtYA/0, /var/lib/elasticsearch/nodes/0/indices/bFsN-CkCSjufBACUL27u0w/0, /var/lib/elasticsearch/nodes/0/indices/mvFmVLaOSN-wAvMgXknzTg/0, /var/lib/elasticsearch/nodes/0/indices/XBSojsDSRs6IXYQNH9j-Jg/0, /var/lib/elasticsearch/nodes/0/indices/0X9qyFynTPac0YZvXn9Ufw/0, /var/lib/elasticsearch/nodes/0/indices/NQbHnD4CTweCU4LgfKR77A/0, /var/lib/elasticsearch/nodes/0/indices/vDe9aUIpQ8ipqbmPEXnTFA/0, /var/lib/elasticsearch/nodes/0/indices/97L23TQ1TJmKHDYzwaYfRg/0, /var/lib/elasticsearch/nodes/0/indices/udZhKWpVSVCAsJnOVcRVLQ/0, /var/lib/elasticsearch/nodes/0/indices/U1b1IEMWRYqzPlDu9TcG0Q/0, /var/lib/elasticsearch/nodes/0/indices/_CLujyNjQgmCxLFBjO79Zg/0, /var/lib/elasticsearch/nodes/0/indices/LNFkjsKnQMCD5yYBHKyyNw/0, /var/lib/elasticsearch/nodes/0/indices/HnFJxvS7TveyGoawLxDFtA/0, /var/lib/elasticsearch/nodes/0/indices/pqkhAZObTj63zylELg7ySQ/0, /var/lib/elasticsearch/nodes/0/indices/61ETZjspQzOk6q5fGn3Twg/0, /var/lib/elasticsearch/nodes/0/indices/5MC-yxWeS6WmubAOIHJt1w/0, /var/lib/elasticsearch/nodes/0/indices/abPk_EJyTluw36cmNf1ZrQ/0, /var/lib/elasticsearch/nodes/0/indices/s-lG4szwR8uMIQfLo_JA1w/0, /var/lib/elasticsearch/nodes/0/indices/Ii5lCjuKRfS5tW4cXEnJng/0, /var/lib/elasticsearch/nodes/0/indices/J6S12mDCR8KGnfWScyiqpw/0, /var/lib/elasticsearch/nodes/0/indices/Sr0g5j7VTzqfcLcd1yoHDQ/0, /var/lib/elasticsearch/nodes/0/indices/8Gk6QC7_TS2p0qRtx6XCmA/0, /var/lib/elasticsearch/nodes/0/indices/BjMiZoZ5SIyBNbwND7fZ5w/0, /var/lib/elasticsearch/nodes/0/indices/pCmnXkAIR6a7k_Kv2uiwig/0, /var/lib/elasticsearch/nodes/0/indices/fSpz7hUCRWuX_YqTOp7FUA/0, /var/lib/elasticsearch/nodes/0/indices/uXXph5eFSPGH7RRuEKDGHQ/0, /var/lib/elasticsearch/nodes/0/indices/nr3MS0J9TzuElqtUjqEwqQ/0, /var/lib/elasticsearch/nodes/0/indices/SNiHj03OT2-YC8wMb7LyLg/0, /var/lib/elasticsearch/nodes/0/indices/pzmAhBOES-KtmESbs0_EKQ/0, /var/lib/elasticsearch/nodes/0/indices/cXhosVdCSayJJrf3tvvG7w/0]. Use 'elasticsearch-node repurpose' tool to clean up at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.6.1.jar:7.6.1] at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.6.1.jar:7.6.1] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.6.1.jar:7.6.1] Caused by: java.lang.IllegalStateException: Node is started with node.data=false, but has shard data: [/var/lib/elasticsearch/nodes/0/indices/e3cMeIVSQdmIwVVgh_Bd0w/0, /var/lib/elasticsearch/nodes/0/indices/Prq1jnKSSoyt3XWJO6iCUg/0, /var/lib/elasticsearch/nodes/0/indices/pnQ2AWwnSca8-hFVL_gtYA/0, /var/lib/elasticsearch/nodes/0/indices/bFsN-CkCSjufBACUL27u0w/0, /var/lib/elasticsearch/nodes/0/indices/mvFmVLaOSN-wAvMgXknzTg/0, /var/lib/elasticsearch/nodes/0/indices/XBSojsDSRs6IXYQNH9j-Jg/0, /var/lib/elasticsearch/nodes/0/indices/0X9qyFynTPac0YZvXn9Ufw/0, /var/lib/elasticsearch/nodes/0/indices/NQbHnD4CTweCU4LgfKR77A/0, /var/lib/elasticsearch/nodes/0/indices/vDe9aUIpQ8ipqbmPEXnTFA/0, /var/lib/elasticsearch/nodes/0/indices/97L23TQ1TJmKHDYzwaYfRg/0, /var/lib/elasticsearch/nodes/0/indices/udZhKWpVSVCAsJnOVcRVLQ/0, /var/lib/elasticsearch/nodes/0/indices/U1b1IEMWRYqzPlDu9TcG0Q/0, /var/lib/elasticsearch/nodes/0/indices/_CLujyNjQgmCxLFBjO79Zg/0, /var/lib/elasticsearch/nodes/0/indices/LNFkjsKnQMCD5yYBHKyyNw/0, /var/lib/elasticsearch/nodes/0/indices/HnFJxvS7TveyGoawLxDFtA/0, /var/lib/elasticsearch/nodes/0/indices/pqkhAZObTj63zylELg7ySQ/0, /var/lib/elasticsearch/nodes/0/indices/61ETZjspQzOk6q5fGn3Twg/0, /var/lib/elasticsearch/nodes/0/indices/5MC-yxWeS6WmubAOIHJt1w/0, /var/lib/elasticsearch/nodes/0/indices/abPk_EJyTluw36cmNf1ZrQ/0, /var/lib/elasticsearch/nodes/0/indices/s-lG4szwR8uMIQfLo_JA1w/0, /var/lib/elasticsearch/nodes/0/indices/Ii5lCjuKRfS5tW4cXEnJng/0, /var/lib/elasticsearch/nodes/0/indices/J6S12mDCR8KGnfWScyiqpw/0, /var/lib/elasticsearch/nodes/0/indices/Sr0g5j7VTzqfcLcd1yoHDQ/0, /var/lib/elasticsearch/nodes/0/indices/8Gk6QC7_TS2p0qRtx6XCmA/0, /var/lib/elasticsearch/nodes/0/indices/BjMiZoZ5SIyBNbwND7fZ5w/0, /var/lib/elasticsearch/nodes/0/indices/pCmnXkAIR6a7k_Kv2uiwig/0, /var/lib/elasticsearch/nodes/0/indices/fSpz7hUCRWuX_YqTOp7FUA/0, /var/lib/elasticsearch/nodes/0/indices/uXXph5eFSPGH7RRuEKDGHQ/0, /var/lib/elasticsearch/nodes/0/indices/nr3MS0J9TzuElqtUjqEwqQ/0, /var/lib/elasticsearch/nodes/0/indices/SNiHj03OT2-YC8wMb7LyLg/0, /var/lib/elasticsearch/nodes/0/indices/pzmAhBOES-KtmESbs0_EKQ/0, /var/lib/elasticsearch/nodes/0/indices/cXhosVdCSayJJrf3tvvG7w/0]. Use 'elasticsearch-node repurpose' tool to clean up at org.elasticsearch.env.NodeEnvironment.ensureNoShardData(NodeEnvironment.java:1083) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:326) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.node.Node.(Node.java:277) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.node.Node.(Node.java:257) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.1.jar:7.6.1] ... 6 more ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 13, 2020, 6:32am UTC](https://discuss.elastic.co/t/add-elasticsearch-node-multiple-machine/223443/6 "2020-03-13T06:32:11Z")
> [@Charles\_Yuliansen](#): > > cluster.initial\_master\_nodes: - x.x.x.246:9300 - x.x.x.99:9301 Why are you listing the node that is not master-eligible as an initial master node? ---
**Author:** ![Charles\_Yuliansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charles_yuliansen/32/45664_2.png) [@Charles\_Yuliansen](https://discuss.elastic.co/u/Charles_Yuliansen)\ **Post date:** [March 13, 2020, 6:37am UTC](https://discuss.elastic.co/t/add-elasticsearch-node-multiple-machine/223443/7 "2020-03-13T06:37:13Z")
hi there thank you for the respond, i'm so blind at this so i follow and try any instruction i found [here](https://stackoverflow.com/questions/55964071/cannot-add-node-to-cluster-elasticsearch/60663846#60663846) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 13, 2020, 6:38am UTC](https://discuss.elastic.co/t/add-elasticsearch-node-multiple-machine/223443/8 "2020-03-13T06:38:59Z")
In that example both nodes are master eligible. What is the size of the cluster you intend to run? ---
**Author:** ![Charles\_Yuliansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charles_yuliansen/32/45664_2.png) [@Charles\_Yuliansen](https://discuss.elastic.co/u/Charles_Yuliansen)\ **Post date:** [March 13, 2020, 6:46am UTC](https://discuss.elastic.co/t/add-elasticsearch-node-multiple-machine/223443/9 "2020-03-13T06:46:10Z")
hi thank you so much for your respond, what does it mean? I want to run a cluster with containing of 1 masternode and 1 data node. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 13, 2020, 6:58am UTC](https://discuss.elastic.co/t/add-elasticsearch-node-multiple-machine/223443/10 "2020-03-13T06:58:37Z")
Why do you want to do that? If you run a single node that is master eligible and holds data it is a single point of failure and you have no high availability. In a cluster with a dedicated master and a dedicated data node the cluster will not be operational if either of the nodes goes down, meaning that you have a less resilient cluster. ---
**Author:** ![Charles\_Yuliansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charles_yuliansen/32/45664_2.png) [@Charles\_Yuliansen](https://discuss.elastic.co/u/Charles_Yuliansen)\ **Post date:** [March 13, 2020, 7:01am UTC](https://discuss.elastic.co/t/add-elasticsearch-node-multiple-machine/223443/11 "2020-03-13T07:01:05Z")
thank you so much for the explanation, i want to expand the size of my running cluster, which is x.x.x.246 as the running cluster. Then what should i do? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 13, 2020, 7:02am UTC](https://discuss.elastic.co/t/add-elasticsearch-node-multiple-machine/223443/12 "2020-03-13T07:02:42Z")
If you are looking for resilience you should always have at least 3 master-eligible nodes in the cluster. The smallest footpring that supports this if typcally one small dedicated master node together with 2 master/data nodes. ---
**Author:** ![Charles\_Yuliansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charles_yuliansen/32/45664_2.png) [@Charles\_Yuliansen](https://discuss.elastic.co/u/Charles_Yuliansen)\ **Post date:** [March 13, 2020, 7:04am UTC](https://discuss.elastic.co/t/add-elasticsearch-node-multiple-machine/223443/13 "2020-03-13T07:04:19Z")
is it means to expanding a cluster size i have to create 3 master node? can you provide me any documentation about cluster expanding node? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 14, 2020, 8:48am UTC](https://discuss.elastic.co/t/add-elasticsearch-node-multiple-machine/223443/14 "2020-03-14T08:48:22Z")
For small clusters you typically do not need dedicated master nodes. Having 3 nodes which are master-eligible and hold data is a good starting ponit. Once you have a cluster set up with one or more nodes [the docs describe how to add more nodes](https://www.elastic.co/guide/en/elasticsearch/reference/current/add-elasticsearch-nodes.html). ---
**Author:** ![Charles\_Yuliansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charles_yuliansen/32/45664_2.png) [@Charles\_Yuliansen](https://discuss.elastic.co/u/Charles_Yuliansen)\ **Post date:** [March 16, 2020, 2:53am UTC](https://discuss.elastic.co/t/add-elasticsearch-node-multiple-machine/223443/15 "2020-03-16T02:53:39Z")
hi there thank you so much for your explanation, what should i do to expand the size of my cluster. In the case it almost full. Should i add master node, or data node. In some articles i saw them make masternode and datanode as true. What does it mean? ---
**Author:** ![Charles\_Yuliansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charles_yuliansen/32/45664_2.png) [@Charles\_Yuliansen](https://discuss.elastic.co/u/Charles_Yuliansen)\ **Post date:** [March 16, 2020, 8:02am UTC](https://discuss.elastic.co/t/add-elasticsearch-node-multiple-machine/223443/16 "2020-03-16T08:02:27Z")
hi there i got update, after few changes it turns out that my data node returning ```auto { "error" : { "root_cause" : [ { "type" : "master_not_discovered_exception", "reason" : null } ], "type" : "master_not_discovered_exception", "reason" : null }, "status" : 503 } ``` conf file ```auto #1 cluster.name: elasticsearch node.name: node-1 node.master : false node.data: true network.host: x.x.x.99 discovery.seed_hosts: ["x.x.x.246"] cluster.initial_master_nodes: ["x.x.x.246"] #2 cluster.name: elasticsearch node.name: Node_master node.master : true node.data: true network.host: x.x.x.246 discovery.seed_hosts: ["x.x.x.99"] cluster.initial_master_nodes: ["x.x.x.246"] ``` whenever i run curl -X GET "X.X.X.99:9200/\_cluster/health?pretty" can you help we what is going on. Thank you ---
**Author:** ![Charles\_Yuliansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charles_yuliansen/32/45664_2.png) [@Charles\_Yuliansen](https://discuss.elastic.co/u/Charles_Yuliansen)\ **Post date:** [March 17, 2020, 8:40am UTC](https://discuss.elastic.co/t/add-elasticsearch-node-multiple-machine/223443/17 "2020-03-17T08:40:39Z")
log from updated conf file `Node_master` `x.x.x.246`: ```auto [root@localhost elasticsearch]# tail -30 elasticsearch.log at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:830) [?:?] [2020-03-17T16:13:56,811][WARN][o.e.c.c.Coordinator] [Node_master] failed to validate incoming join request from node [{node-1}{jb_3lJq1R5-BZtxlPs_NyQ}{wXqAiv4eQ-2lfI_x7hHT_w}{10.64.2.99}{10.64.2.99:9300}{dil}{ml.machine_memory=8201347072, ml.max_open_jobs=20, xpack.installed=true}] org.elasticsearch.transport.RemoteTransportException: [node-1][x.x.x.99:9300][internal:cluster/coordination/join/validate] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid P4QlwvuRRGSmlT77RroSjA than local cluster uuid oUoIe2-bSbS2UPg722ud9Q, rejecting at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) ~[?:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:315) ~[?:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.1.jar:7.6.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:830) [?:?] [2020-03-17T16:13:57,812][WARN][o.e.c.c.Coordinator] [Node_master] failed to validate incoming join request from node [{node-1}{jb_3lJq1R5-BZtxlPs_NyQ}{wXqAiv4eQ-2lfI_x7hHT_w}{x.x.x.99}{x.x.x.99:9300}{dil}{ml.machine_memory=8201347072, ml.max_open_jobs=20, xpack.installed=true}] org.elasticsearch.transport.RemoteTransportException: [node-1][x.x.x.99:9300][internal:cluster/coordination/join/validate] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid P4QlwvuRRGSmlT77RroSjA than local cluster uuid oUoIe2-bSbS2UPg722ud9Q, rejecting at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) ~[?:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:315) ~[?:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.1.jar:7.6.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:830) [?:?] ``` `node-1` x.x.x.99 : ```auto [root@dev elasticsearch]# tail -30 elasticsearch.log at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.1.jar:7.6.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:830) ~[?:?] [2020-03-17T15:33:12,707][INFO][o.e.c.c.JoinHelper] [node-1] failed to join {Node_master}{0UHYehfNQ2-WCadTC_VVkA}{t85CF-LWTSav7RGx_dIldQ}{x.x.x.246}{x.x.x.246:9300}{dilm}{ml.machine_memory=1907810304, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={node-1}{jb_3lJq1R5-BZtxlPs_NyQ}{wXqAiv4eQ-2lfI_x7hHT_w}{x.x.x.99}{1x.x.x.99:9300}{dil}{ml.machine_memory=8201347072, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=177, lastAcceptedTerm=8, lastAcceptedVersion=100, sourceNode={node-1}{jb_3lJq1R5-BZtxlPs_NyQ}{wXqAiv4eQ-2lfI_x7hHT_w}{10.64.2.99}{10.64.2.99:9300}{dil}{ml.machine_memory=8201347072, xpack.installed=true, ml.max_open_jobs=20}, targetNode={Node_master}{0UHYehfNQ2-WCadTC_VVkA}{t85CF-LWTSav7RGx_dIldQ}{x.x.x.246}{x.x.x.246:9300}{dilm}{ml.machine_memory=1907810304, ml.max_open_jobs=20, xpack.installed=true}}]} org.elasticsearch.transport.RemoteTransportException: [Node_master][10.64.2.246:9300][internal:cluster/coordination/join] Caused by: java.lang.IllegalStateException: failure when sending a validation request to node at org.elasticsearch.cluster.coordination.Coordinator$2.onFailure(Coordinator.java:514) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1118) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1118) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:244) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) ~[elasticsearch-7.6.1.jar:7.6.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:830) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [node-1][10.64.2.99:9300][internal:cluster/coordination/join/validate] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid P4QlwvuRRGSmlT77RroSjA than local cluster uuid oUoIe2-bSbS2UPg722ud9Q, rejecting at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) ~[?:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:315) ~[?:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) ~[elasticsearch-7.6.1.jar:7.6.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.1.jar:7.6.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:830) ~[?:?] ``` both on running state ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 14, 2020, 8:40am UTC](https://discuss.elastic.co/t/add-elasticsearch-node-multiple-machine/223443/18 "2020-04-14T08:40:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.