# Recreation of ES data node failed **URL:** \ **Category:** Elasticsearch\ **Created:** [September 6, 2020, 11:30am UTC](https://discuss.elastic.co/t/recreation-of-es-data-node-failed/247669 "2020-09-06T11:30:05Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![KOTOXJle6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kotoxjle6/32/131136_2.png) [@KOTOXJle6](https://discuss.elastic.co/u/KOTOXJle6)\ **Post date:** [September 6, 2020, 11:30am UTC](https://discuss.elastic.co/t/recreation-of-es-data-node-failed/247669/1 "2020-09-06T11:30:05Z")
Good afternoon. There was an elasticsearch cluster consisting of three data nodes and one coordinator with Kibana. It happened that one server died and was re-created. The ES version is the same, Java is the same, and the configuration in elasticsearch. yml is the same. The service started, there are no errors, but this node did not appear in cluster monitoring. curl localhost:9200/\_cat/nodes/ doesnt show problem node. ![photo_2020-09-06_14-15-17](https://us1.discourse-cdn.com/elastic/original/3X/8/0/803671105b567f972702770bab82c8e514ac98de.jpeg) Please tell me what the problem may be? ---
**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\ **Post date:** [September 6, 2020, 11:39am UTC](https://discuss.elastic.co/t/recreation-of-es-data-node-failed/247669/2 "2020-09-06T11:39:39Z")
Can you share log file from the started node ? ---
**Author:** ![KOTOXJle6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kotoxjle6/32/131136_2.png) [@KOTOXJle6](https://discuss.elastic.co/u/KOTOXJle6)\ **Post date:** [September 6, 2020, 11:47am UTC](https://discuss.elastic.co/t/recreation-of-es-data-node-failed/247669/3 "2020-09-06T11:47:32Z")
> [@KOTOXJle6](#): > > doesnt show problem node. Cant put full log from /var/log/elasticsearch/cluster.log, its too big. ``` Caused by: java.lang.IllegalStateException: failure when sending a validation request to node at org.elasticsearch.cluster.coordination.Coordinator$3.onFailure(Coordinator.java:500) ~[elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.cluster.coordination.JoinHelper$5.handleException(JoinHelper.java:359) ~[elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[elasticsearch-7.1.0.jar:7.1.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_211] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_211] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211] Caused by: org.elasticsearch.transport.RemoteTransportException: [elk-es-node-02][10.199.5.104:9300][internal:cluster/coordination/join/validate] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid VdZPYWRCT8eMvjFOpWB6Lw than local cluster uuid 693EJRStTo-3GwqanfvZkA, rejecting at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:147) ~[elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251) ~[?:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309) ~[?:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.1.0.jar:7.1.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_211] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_211] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_211] [2020-09-06T14:51:13,609][INFO][o.e.c.c.JoinHelper] [elk-es-node-02] failed to join {elk-es-node-03}{sPaS62bdS8CJrY9MWNUkgQ}{eToptbY8TFahH6E1iv_RLQ}{10.199.5.105}{10.199.5.105:9300}{ml.machine_memory=12592672768, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elk-es-node-02}{u2qN94UbTt6hHKl_ShftVw}{wsLKW5uGTUSnM6_YjvmxIQ}{10.199.5.104}{10.199.5.104:9300}{ml.machine_memory=12564250624, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional.empty} org.elasticsearch.transport.RemoteTransportException: [elk-es-node-03][10.199.5.105:9300][internal:cluster/coordination/join] Caused by: java.lang.IllegalStateException: failure when sending a validation request to node at org.elasticsearch.cluster.coordination.Coordinator$3.onFailure(Coordinator.java:500) ~[elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.cluster.coordination.JoinHelper$5.handleException(JoinHelper.java:359) ~[elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[elasticsearch-7.1.0.jar:7.1.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_211] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_211] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211] Caused by: org.elasticsearch.transport.RemoteTransportException: [elk-es-node-02][10.199.5.104:9300][internal:cluster/coordination/join/validate] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid VdZPYWRCT8eMvjFOpWB6Lw than local cluster uuid 693EJRStTo-3GwqanfvZkA, rejecting at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:147) ~[elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251) ~[?:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309) ~[?:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.1.0.jar:7.1.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_211] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_211] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_211] [2020-09-06T14:51:14,115][WARN][r.suppressed] [elk-es-node-02] path: /_license, params: {} org.elasticsearch.discovery.MasterNotDiscoveredException: null at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:259) [elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:322) [elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:249) [elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:555) [elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.1.0.jar:7.1.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_211] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_211] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211] [2020-09-06T14:51:14,356][WARN][r.suppressed] [elk-es-node-02] path: /_license, params: {} org.elasticsearch.discovery.MasterNotDiscoveredException: null at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:259) [elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:322) [elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:249) [elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:555) [elasticsearch-7.1.0.jar:7.1.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.1.0.jar:7.1.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_211] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_211] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211] ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 6, 2020, 11:55am UTC](https://discuss.elastic.co/t/recreation-of-es-data-node-failed/247669/4 "2020-09-06T11:55:29Z")
> [@KOTOXJle6](#): > > `[2020-09-06T13:47:56,386][WARN][o.e.b.BootstrapChecks] [elk-es-node-02] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured` It does not seem like the new node is configured to link up with the other nodes. What does your elasticsearch.yml config file look like? ---
**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\ **Post date:** [September 6, 2020, 11:56am UTC](https://discuss.elastic.co/t/recreation-of-es-data-node-failed/247669/5 "2020-09-06T11:56:36Z")
Looks a single node and not part of a cluster Make sure you use same cluster config from your existing setup ---
**Author:** ![KOTOXJle6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kotoxjle6/32/131136_2.png) [@KOTOXJle6](https://discuss.elastic.co/u/KOTOXJle6)\ **Post date:** [September 6, 2020, 12:00pm UTC](https://discuss.elastic.co/t/recreation-of-es-data-node-failed/247669/6 "2020-09-06T12:00:57Z")
Sorry, i've post elasticsearch.log at first time. Now its updated to cluster.log. ---
**Author:** ![KOTOXJle6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kotoxjle6/32/131136_2.png) [@KOTOXJle6](https://discuss.elastic.co/u/KOTOXJle6)\ **Post date:** [September 6, 2020, 12:20pm UTC](https://discuss.elastic.co/t/recreation-of-es-data-node-failed/247669/7 "2020-09-06T12:20:28Z")
Looks like the problem in this "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid VdZPYWRCT8eMvjFOpWB6Lw than local cluster uuid 693EJRStTo-3GwqanfvZkA, rejecting". But i dont understand how that can be fixed. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 6, 2020, 12:42pm UTC](https://discuss.elastic.co/t/recreation-of-es-data-node-failed/247669/8 "2020-09-06T12:42:01Z")
I think the new node has been started up without the correct config and created a cluster of its own. Wipe the storage of the new node and launch it with the correct config as that should allow it to join. ---
**Author:** ![KOTOXJle6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kotoxjle6/32/131136_2.png) [@KOTOXJle6](https://discuss.elastic.co/u/KOTOXJle6)\ **Post date:** [September 6, 2020, 12:47pm UTC](https://discuss.elastic.co/t/recreation-of-es-data-node-failed/247669/9 "2020-09-06T12:47:09Z")
It worked! I realized that the problem was related to the fact that I first started ES with default configuration to check .conf, which caused a new cluster to be created, and then tried to attach the node to the old cluster and got an error. Thanks all! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 4, 2020, 12:47pm UTC](https://discuss.elastic.co/t/recreation-of-es-data-node-failed/247669/10 "2020-10-04T12:47:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.