Add elasticsearch node multiple machine

Hi, im currently doing some poc about add node to elasticsearch cluster with multiple machine. The master node should be x.x.x.246 and the data node that i want to add is x.x.x.99. Running the latest elasticsearch 7.6. I've make sure that both elastic are on the same version.

Here is my config file for master node.

cluster.name : elasticsearch
node.name : Node_master
node.master : true
node.data : false
network.host : 10.64.2.246
http.port : 9200
transport.tcp.port: 9300
discovery.seed_hosts:
   - x.x.x.246:9300
   - x.x.x.99:9301
cluster.initial_master_nodes:
   - x.x.x.246:9300
   - x.x.x.99:9301

here is my config for data node

cluster.name : elasticsearch
node.name : Node-1
node.master : false
node.data : true
network.host : 10.64.2.99
http.port : 9201
transport.tcp.port: 9301
discovery.seed_hosts:
   - x.x.x.246:9300
   - x.x.x.99:9301
cluster.initial_master_nodes:
   - x.x.x.246:9300
   - x.x.x.99:9301

here is the error returned when i restart elasticsearch on master node :

[root@localhost elasticsearch]# service elasticsearch start
Starting elasticsearch (via systemctl):  Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
                                                           [FAILED]

systemctl view on master node:

[root@localhost elasticsearch]# systemctl status elasticsearch.service -l
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2020-03-13 11:37:44 WIB; 2min 20s ago
     Docs: http://www.elastic.co
  Process: 127072 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 127072 (code=exited, status=1/FAILURE)

Mar 13 11:37:43 localhost.localdomain elasticsearch[127072]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Mar 13 11:37:43 localhost.localdomain elasticsearch[127072]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
Mar 13 11:37:43 localhost.localdomain elasticsearch[127072]: at org.elasticsearch.cli.Command.main(Command.java:90)
Mar 13 11:37:43 localhost.localdomain elasticsearch[127072]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
Mar 13 11:37:43 localhost.localdomain elasticsearch[127072]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
Mar 13 11:37:43 localhost.localdomain elasticsearch[127072]: For complete error details, refer to the log at /var/log/elasticsearch/elasticsearch.log
Mar 13 11:37:44 localhost.localdomain systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Mar 13 11:37:44 localhost.localdomain systemd[1]: Failed to start Elasticsearch.
Mar 13 11:37:44 localhost.localdomain systemd[1]: Unit elasticsearch.service entered failed state.
Mar 13 11:37:44 localhost.localdomain systemd[1]: elasticsearch.service failed.

here is the error returned when i restart elasticsearch on data node :

[root@dev elasticsearch]# service elasticsearch restart
Restarting elasticsearch (via systemctl):  Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
                                                           [FAILED]
[root@dev elasticsearch]# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2020-03-13 10:29:21 WIB; 16s ago
     Docs: http://www.elastic.co
  Process: 10474 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 10474 (code=exited, status=1/FAILURE)

systemctl view on data node:

● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2020-03-13 10:29:21 WIB; 15min ago
     Docs: http://www.elastic.co
  Process: 10474 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 10474 (code=exited, status=1/FAILURE)

Mar 13 10:29:21 dev.int.cbn.net.id elasticsearch[10474]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
Mar 13 10:29:21 dev.int.cbn.net.id elasticsearch[10474]: at org.elasticsearch.cli.Command.main(Command.java:90)
Mar 13 10:29:21 dev.int.cbn.net.id elasticsearch[10474]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
Mar 13 10:29:21 dev.int.cbn.net.id elasticsearch[10474]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
Mar 13 10:29:21 dev.int.cbn.net.id elasticsearch[10474]: Caused by: com.fasterxml.jackson.dataformat.yaml.snakeyaml.error.MarkedYAMLException: while scanning a simple key
Mar 13 10:29:21 dev.int.cbn.net.id elasticsearch[10474]: in 'reader', line 25, column 1:
Mar 13 10:29:21 dev.int.cbn.net.id systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Mar 13 10:29:21 dev.int.cbn.net.id systemd[1]: Failed to start Elasticsearch.
Mar 13 10:29:21 dev.int.cbn.net.id systemd[1]: Unit elasticsearch.service entered failed state.
Mar 13 10:29:21 dev.int.cbn.net.id systemd[1]: elasticsearch.service failed.

any answer or explanation will be appreciated so much. Thank you

Can you check that and post the output?

1 Like

hi there, thank you for your respond.
here is the log at master node x.x.x.246:

[root@localhost elasticsearch]# tail -l elasticsearch.log -l
tail: invalid option -- 'l'
Try 'tail --help' for more information.
[root@localhost elasticsearch]# tail -l elasticsearch.log
Caused by: java.lang.IllegalStateException: Node is started with node.data=fa   lse, but has shard data: [/var/lib/elasticsearch/nodes/0/indices/e3cMeIVSQdmI   wVVgh_Bd0w/0, /var/lib/elasticsearch/nodes/0/indices/Prq1jnKSSoyt3XWJO6iCUg/0   , /var/lib/elasticsearch/nodes/0/indices/pnQ2AWwnSca8-hFVL_gtYA/0, /var/lib/e   lasticsearch/nodes/0/indices/bFsN-CkCSjufBACUL27u0w/0, /var/lib/elasticsearch   /nodes/0/indices/mvFmVLaOSN-wAvMgXknzTg/0, /var/lib/elasticsearch/nodes/0/ind   ices/XBSojsDSRs6IXYQNH9j-Jg/0, /var/lib/elasticsearch/nodes/0/indices/0X9qyFy   nTPac0YZvXn9Ufw/0, /var/lib/elasticsearch/nodes/0/indices/NQbHnD4CTweCU4LgfKR   77A/0, /var/lib/elasticsearch/nodes/0/indices/vDe9aUIpQ8ipqbmPEXnTFA/0, /var/   lib/elasticsearch/nodes/0/indices/97L23TQ1TJmKHDYzwaYfRg/0, /var/lib/elastics   earch/nodes/0/indices/udZhKWpVSVCAsJnOVcRVLQ/0, /var/lib/elasticsearch/nodes/   0/indices/U1b1IEMWRYqzPlDu9TcG0Q/0, /var/lib/elasticsearch/nodes/0/indices/_C   LujyNjQgmCxLFBjO79Zg/0, /var/lib/elasticsearch/nodes/0/indices/LNFkjsKnQMCD5y   YBHKyyNw/0, /var/lib/elasticsearch/nodes/0/indices/HnFJxvS7TveyGoawLxDFtA/0,    /var/lib/elasticsearch/nodes/0/indices/pqkhAZObTj63zylELg7ySQ/0, /var/lib/ela   sticsearch/nodes/0/indices/61ETZjspQzOk6q5fGn3Twg/0, /var/lib/elasticsearch/n   odes/0/indices/5MC-yxWeS6WmubAOIHJt1w/0, /var/lib/elasticsearch/nodes/0/indic   es/abPk_EJyTluw36cmNf1ZrQ/0, /var/lib/elasticsearch/nodes/0/indices/s-lG4szwR   8uMIQfLo_JA1w/0, /var/lib/elasticsearch/nodes/0/indices/Ii5lCjuKRfS5tW4cXEnJn   g/0, /var/lib/elasticsearch/nodes/0/indices/J6S12mDCR8KGnfWScyiqpw/0, /var/li   b/elasticsearch/nodes/0/indices/Sr0g5j7VTzqfcLcd1yoHDQ/0, /var/lib/elasticsea   rch/nodes/0/indices/8Gk6QC7_TS2p0qRtx6XCmA/0, /var/lib/elasticsearch/nodes/0/   indices/BjMiZoZ5SIyBNbwND7fZ5w/0, /var/lib/elasticsearch/nodes/0/indices/pCmn   XkAIR6a7k_Kv2uiwig/0, /var/lib/elasticsearch/nodes/0/indices/fSpz7hUCRWuX_YqT   Op7FUA/0, /var/lib/elasticsearch/nodes/0/indices/uXXph5eFSPGH7RRuEKDGHQ/0, /v   ar/lib/elasticsearch/nodes/0/indices/nr3MS0J9TzuElqtUjqEwqQ/0, /var/lib/elast   icsearch/nodes/0/indices/SNiHj03OT2-YC8wMb7LyLg/0, /var/lib/elasticsearch/nod   es/0/indices/pzmAhBOES-KtmESbs0_EKQ/0, /var/lib/elasticsearch/nodes/0/indices   /cXhosVdCSayJJrf3tvvG7w/0]. Use 'elasticsearch-node repurpose' tool to clean    up
        at org.elasticsearch.env.NodeEnvironment.ensureNoShardData(NodeEnviro   nment.java:1083) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:   326) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-   7.6.1.jar:7.6.1]
        at org.elasticsearch.node.Node.<init>(Node.java:257) ~[elasticsearch-   7.6.1.jar:7.6.1]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221)    ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[   elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[e   lasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:   170) ~[elasticsearch-7.6.1.jar:7.6.1]
        ... 6 more

here is the log at data node x.x.x.99:

[root@dev elasticsearch]# tail -l elasticsearch.log
        ... 3 more
[2020-03-13T09:47:27,870][INFO ][o.e.c.r.a.AllocationService] [dev.int.cbn.net.id] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[g_station][0]]]).
[2020-03-13T10:29:18,483][INFO ][o.e.n.Node               ] [dev.int.cbn.net.id] stopping ...
[2020-03-13T10:29:18,566][INFO ][o.e.x.w.WatcherService   ] [dev.int.cbn.net.id] stopping watch service, reason [shutdown initiated]
[2020-03-13T10:29:18,569][INFO ][o.e.x.w.WatcherLifeCycleService] [dev.int.cbn.net.id] watcher has stopped and shutdown
[2020-03-13T10:29:18,789][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [dev.int.cbn.net.id] [controller/5534] [Main.cc@150] Ml controller exiting
[2020-03-13T10:29:18,795][INFO ][o.e.x.m.p.NativeController] [dev.int.cbn.net.id] Native controller process has stopped - no new native processes can be started
[2020-03-13T10:29:18,948][INFO ][o.e.n.Node               ] [dev.int.cbn.net.id] stopped
[2020-03-13T10:29:18,949][INFO ][o.e.n.Node               ] [dev.int.cbn.net.id] closing ...
[2020-03-13T10:29:19,004][INFO ][o.e.n.Node               ] [dev.int.cbn.net.id] closed

Can you show a bit more than that please, there's not enough on either node to show what is or isn't happening.

hi im sorry about that.
here is the data node:

[root@dev elasticsearch]# tail -20 elasticsearch.log
        at org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:180) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:104) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:95) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:106) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:45) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.lambda$asyncShardOperation$0(TransportSingleShardAction.java:110) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.1.jar:7.6.1]
        ... 3 more
[2020-03-13T09:47:27,870][INFO ][o.e.c.r.a.AllocationService] [dev.int.cbn.net.id] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[g_station][0]]]).
[2020-03-13T10:29:18,483][INFO ][o.e.n.Node               ] [dev.int.cbn.net.id] stopping ...
[2020-03-13T10:29:18,566][INFO ][o.e.x.w.WatcherService   ] [dev.int.cbn.net.id] stopping watch service, reason [shutdown initiated]
[2020-03-13T10:29:18,569][INFO ][o.e.x.w.WatcherLifeCycleService] [dev.int.cbn.net.id] watcher has stopped and shutdown
[2020-03-13T10:29:18,789][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [dev.int.cbn.net.id] [controller/5534] [Main.cc@150] Ml controller exiting
[2020-03-13T10:29:18,795][INFO ][o.e.x.m.p.NativeController] [dev.int.cbn.net.id] Native controller process has stopped - no new native processes can be started
[2020-03-13T10:29:18,948][INFO ][o.e.n.Node               ] [dev.int.cbn.net.id] stopped
[2020-03-13T10:29:18,949][INFO ][o.e.n.Node               ] [dev.int.cbn.net.id] closing ...
[2020-03-13T10:29:19,004][INFO ][o.e.n.Node               ] [dev.int.cbn.net.id] closed

here is the master node :

[root@localhost elasticsearch]# tail -20 elasticsearch.log
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.6.1.jar:7.6.1]
[2020-03-13T12:07:51,762][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [Node_master] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Node is started with node.data=false, but has shard data: [/var/lib/elasticsearch/nodes/0/indices/e3cMeIVSQdmIwVVgh_Bd0w/0, /var/lib/elasticsearch/nodes/0/indices/Prq1jnKSSoyt3XWJO6iCUg/0, /var/lib/elasticsearch/nodes/0/indices/pnQ2AWwnSca8-hFVL_gtYA/0, /var/lib/elasticsearch/nodes/0/indices/bFsN-CkCSjufBACUL27u0w/0, /var/lib/elasticsearch/nodes/0/indices/mvFmVLaOSN-wAvMgXknzTg/0, /var/lib/elasticsearch/nodes/0/indices/XBSojsDSRs6IXYQNH9j-Jg/0, /var/lib/elasticsearch/nodes/0/indices/0X9qyFynTPac0YZvXn9Ufw/0, /var/lib/elasticsearch/nodes/0/indices/NQbHnD4CTweCU4LgfKR77A/0, /var/lib/elasticsearch/nodes/0/indices/vDe9aUIpQ8ipqbmPEXnTFA/0, /var/lib/elasticsearch/nodes/0/indices/97L23TQ1TJmKHDYzwaYfRg/0, /var/lib/elasticsearch/nodes/0/indices/udZhKWpVSVCAsJnOVcRVLQ/0, /var/lib/elasticsearch/nodes/0/indices/U1b1IEMWRYqzPlDu9TcG0Q/0, /var/lib/elasticsearch/nodes/0/indices/_CLujyNjQgmCxLFBjO79Zg/0, /var/lib/elasticsearch/nodes/0/indices/LNFkjsKnQMCD5yYBHKyyNw/0, /var/lib/elasticsearch/nodes/0/indices/HnFJxvS7TveyGoawLxDFtA/0, /var/lib/elasticsearch/nodes/0/indices/pqkhAZObTj63zylELg7ySQ/0, /var/lib/elasticsearch/nodes/0/indices/61ETZjspQzOk6q5fGn3Twg/0, /var/lib/elasticsearch/nodes/0/indices/5MC-yxWeS6WmubAOIHJt1w/0, /var/lib/elasticsearch/nodes/0/indices/abPk_EJyTluw36cmNf1ZrQ/0, /var/lib/elasticsearch/nodes/0/indices/s-lG4szwR8uMIQfLo_JA1w/0, /var/lib/elasticsearch/nodes/0/indices/Ii5lCjuKRfS5tW4cXEnJng/0, /var/lib/elasticsearch/nodes/0/indices/J6S12mDCR8KGnfWScyiqpw/0, /var/lib/elasticsearch/nodes/0/indices/Sr0g5j7VTzqfcLcd1yoHDQ/0, /var/lib/elasticsearch/nodes/0/indices/8Gk6QC7_TS2p0qRtx6XCmA/0, /var/lib/elasticsearch/nodes/0/indices/BjMiZoZ5SIyBNbwND7fZ5w/0, /var/lib/elasticsearch/nodes/0/indices/pCmnXkAIR6a7k_Kv2uiwig/0, /var/lib/elasticsearch/nodes/0/indices/fSpz7hUCRWuX_YqTOp7FUA/0, /var/lib/elasticsearch/nodes/0/indices/uXXph5eFSPGH7RRuEKDGHQ/0, /var/lib/elasticsearch/nodes/0/indices/nr3MS0J9TzuElqtUjqEwqQ/0, /var/lib/elasticsearch/nodes/0/indices/SNiHj03OT2-YC8wMb7LyLg/0, /var/lib/elasticsearch/nodes/0/indices/pzmAhBOES-KtmESbs0_EKQ/0, /var/lib/elasticsearch/nodes/0/indices/cXhosVdCSayJJrf3tvvG7w/0]. Use 'elasticsearch-node repurpose' tool to clean up
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.6.1.jar:7.6.1]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.6.1.jar:7.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.6.1.jar:7.6.1]
Caused by: java.lang.IllegalStateException: Node is started with node.data=false, but has shard data: [/var/lib/elasticsearch/nodes/0/indices/e3cMeIVSQdmIwVVgh_Bd0w/0, /var/lib/elasticsearch/nodes/0/indices/Prq1jnKSSoyt3XWJO6iCUg/0, /var/lib/elasticsearch/nodes/0/indices/pnQ2AWwnSca8-hFVL_gtYA/0, /var/lib/elasticsearch/nodes/0/indices/bFsN-CkCSjufBACUL27u0w/0, /var/lib/elasticsearch/nodes/0/indices/mvFmVLaOSN-wAvMgXknzTg/0, /var/lib/elasticsearch/nodes/0/indices/XBSojsDSRs6IXYQNH9j-Jg/0, /var/lib/elasticsearch/nodes/0/indices/0X9qyFynTPac0YZvXn9Ufw/0, /var/lib/elasticsearch/nodes/0/indices/NQbHnD4CTweCU4LgfKR77A/0, /var/lib/elasticsearch/nodes/0/indices/vDe9aUIpQ8ipqbmPEXnTFA/0, /var/lib/elasticsearch/nodes/0/indices/97L23TQ1TJmKHDYzwaYfRg/0, /var/lib/elasticsearch/nodes/0/indices/udZhKWpVSVCAsJnOVcRVLQ/0, /var/lib/elasticsearch/nodes/0/indices/U1b1IEMWRYqzPlDu9TcG0Q/0, /var/lib/elasticsearch/nodes/0/indices/_CLujyNjQgmCxLFBjO79Zg/0, /var/lib/elasticsearch/nodes/0/indices/LNFkjsKnQMCD5yYBHKyyNw/0, /var/lib/elasticsearch/nodes/0/indices/HnFJxvS7TveyGoawLxDFtA/0, /var/lib/elasticsearch/nodes/0/indices/pqkhAZObTj63zylELg7ySQ/0, /var/lib/elasticsearch/nodes/0/indices/61ETZjspQzOk6q5fGn3Twg/0, /var/lib/elasticsearch/nodes/0/indices/5MC-yxWeS6WmubAOIHJt1w/0, /var/lib/elasticsearch/nodes/0/indices/abPk_EJyTluw36cmNf1ZrQ/0, /var/lib/elasticsearch/nodes/0/indices/s-lG4szwR8uMIQfLo_JA1w/0, /var/lib/elasticsearch/nodes/0/indices/Ii5lCjuKRfS5tW4cXEnJng/0, /var/lib/elasticsearch/nodes/0/indices/J6S12mDCR8KGnfWScyiqpw/0, /var/lib/elasticsearch/nodes/0/indices/Sr0g5j7VTzqfcLcd1yoHDQ/0, /var/lib/elasticsearch/nodes/0/indices/8Gk6QC7_TS2p0qRtx6XCmA/0, /var/lib/elasticsearch/nodes/0/indices/BjMiZoZ5SIyBNbwND7fZ5w/0, /var/lib/elasticsearch/nodes/0/indices/pCmnXkAIR6a7k_Kv2uiwig/0, /var/lib/elasticsearch/nodes/0/indices/fSpz7hUCRWuX_YqTOp7FUA/0, /var/lib/elasticsearch/nodes/0/indices/uXXph5eFSPGH7RRuEKDGHQ/0, /var/lib/elasticsearch/nodes/0/indices/nr3MS0J9TzuElqtUjqEwqQ/0, /var/lib/elasticsearch/nodes/0/indices/SNiHj03OT2-YC8wMb7LyLg/0, /var/lib/elasticsearch/nodes/0/indices/pzmAhBOES-KtmESbs0_EKQ/0, /var/lib/elasticsearch/nodes/0/indices/cXhosVdCSayJJrf3tvvG7w/0]. Use 'elasticsearch-node repurpose' tool to clean up
        at org.elasticsearch.env.NodeEnvironment.ensureNoShardData(NodeEnvironment.java:1083) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:326) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.node.Node.<init>(Node.java:257) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.1.jar:7.6.1]
        ... 6 more

Why are you listing the node that is not master-eligible as an initial master node?

hi there thank you for the respond, i'm so blind at this so i follow and try any instruction i found here

In that example both nodes are master eligible. What is the size of the cluster you intend to run?

1 Like

hi thank you so much for your respond, what does it mean? I want to run a cluster with containing of 1 masternode and 1 data node.

Why do you want to do that? If you run a single node that is master eligible and holds data it is a single point of failure and you have no high availability. In a cluster with a dedicated master and a dedicated data node the cluster will not be operational if either of the nodes goes down, meaning that you have a less resilient cluster.

thank you so much for the explanation, i want to expand the size of my running cluster, which is x.x.x.246 as the running cluster. Then what should i do?

If you are looking for resilience you should always have at least 3 master-eligible nodes in the cluster. The smallest footpring that supports this if typcally one small dedicated master node together with 2 master/data nodes.

1 Like

is it means to expanding a cluster size i have to create 3 master node? can you provide me any documentation about cluster expanding node?

For small clusters you typically do not need dedicated master nodes. Having 3 nodes which are master-eligible and hold data is a good starting ponit. Once you have a cluster set up with one or more nodes the docs describe how to add more nodes.

1 Like

hi there thank you so much for your explanation, what should i do to expand the size of my cluster. In the case it almost full. Should i add master node, or data node. In some articles i saw them make masternode and datanode as true. What does it mean?

hi there i got update, after few changes it turns out that my data node returning

{
  "error" : {
    "root_cause" : [
      {
        "type" : "master_not_discovered_exception",
        "reason" : null
      }
    ],
    "type" : "master_not_discovered_exception",
    "reason" : null
  },
  "status" : 503
}

conf file

#1
cluster.name: elasticsearch
node.name: node-1
node.master : false
node.data: true
network.host: x.x.x.99
discovery.seed_hosts: ["x.x.x.246"]
cluster.initial_master_nodes: ["x.x.x.246"]
#2
cluster.name: elasticsearch
node.name: Node_master
node.master : true
node.data: true
network.host: x.x.x.246
discovery.seed_hosts: ["x.x.x.99"]
cluster.initial_master_nodes: ["x.x.x.246"]

whenever i run curl -X GET "X.X.X.99:9200/_cluster/health?pretty" can you help we what is going on. Thank you

log from updated conf file
Node_master x.x.x.246:

[root@localhost elasticsearch]# tail -30 elasticsearch.log
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:830) [?:?]
[2020-03-17T16:13:56,811][WARN ][o.e.c.c.Coordinator      ] [Node_master] failed to validate incoming join request from node [{node-1}{jb_3lJq1R5-BZtxlPs_NyQ}{wXqAiv4eQ-2lfI_x7hHT_w}{10.64.2.99}{10.64.2.99:9300}{dil}{ml.machine_memory=8201347072, ml.max_open_jobs=20, xpack.installed=true}]
org.elasticsearch.transport.RemoteTransportException: [node-1][x.x.x.99:9300][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid P4QlwvuRRGSmlT77RroSjA than local cluster uuid oUoIe2-bSbS2UPg722ud9Q, rejecting
        at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) ~[?:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:315) ~[?:?]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.1.jar:7.6.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:830) [?:?]
[2020-03-17T16:13:57,812][WARN ][o.e.c.c.Coordinator      ] [Node_master] failed to validate incoming join request from node [{node-1}{jb_3lJq1R5-BZtxlPs_NyQ}{wXqAiv4eQ-2lfI_x7hHT_w}{x.x.x.99}{x.x.x.99:9300}{dil}{ml.machine_memory=8201347072, ml.max_open_jobs=20, xpack.installed=true}]
org.elasticsearch.transport.RemoteTransportException: [node-1][x.x.x.99:9300][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid P4QlwvuRRGSmlT77RroSjA than local cluster uuid oUoIe2-bSbS2UPg722ud9Q, rejecting
        at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) ~[?:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:315) ~[?:?]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.1.jar:7.6.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:830) [?:?]

node-1 x.x.x.99 :

[root@dev elasticsearch]# tail -30 elasticsearch.log
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.1.jar:7.6.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:830) ~[?:?]
[2020-03-17T15:33:12,707][INFO ][o.e.c.c.JoinHelper       ] [node-1] failed to join {Node_master}{0UHYehfNQ2-WCadTC_VVkA}{t85CF-LWTSav7RGx_dIldQ}{x.x.x.246}{x.x.x.246:9300}{dilm}{ml.machine_memory=1907810304, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={node-1}{jb_3lJq1R5-BZtxlPs_NyQ}{wXqAiv4eQ-2lfI_x7hHT_w}{x.x.x.99}{1x.x.x.99:9300}{dil}{ml.machine_memory=8201347072, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=177, lastAcceptedTerm=8, lastAcceptedVersion=100, sourceNode={node-1}{jb_3lJq1R5-BZtxlPs_NyQ}{wXqAiv4eQ-2lfI_x7hHT_w}{10.64.2.99}{10.64.2.99:9300}{dil}{ml.machine_memory=8201347072, xpack.installed=true, ml.max_open_jobs=20}, targetNode={Node_master}{0UHYehfNQ2-WCadTC_VVkA}{t85CF-LWTSav7RGx_dIldQ}{x.x.x.246}{x.x.x.246:9300}{dilm}{ml.machine_memory=1907810304, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [Node_master][10.64.2.246:9300][internal:cluster/coordination/join]
Caused by: java.lang.IllegalStateException: failure when sending a validation request to node
        at org.elasticsearch.cluster.coordination.Coordinator$2.onFailure(Coordinator.java:514) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1118) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1118) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:244) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) ~[elasticsearch-7.6.1.jar:7.6.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-1][10.64.2.99:9300][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid P4QlwvuRRGSmlT77RroSjA than local cluster uuid oUoIe2-bSbS2UPg722ud9Q, rejecting
        at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) ~[?:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:315) ~[?:?]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.1.jar:7.6.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:830) ~[?:?]

both on running state