Elastic search Node Failure

HI,

I am trying to start elasticsearch and it gives me the following error.

[2019-12-12T09:36:49,083][WARN ][o.e.b.BootstrapChecks ] [analytics-node] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2019-12-12T09:36:49,096][INFO ][o.e.c.c.Coordinator ] [analytics-node] cluster UUID [qFf9Z6ZTS3qiCHJuzl3yog]
[2019-12-12T09:36:49,125][INFO ][o.e.c.c.ClusterBootstrapService] [analytics-node] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2019-12-12T09:36:49,372][INFO ][o.e.c.s.MasterService ] [analytics-node] elected-as-master ([1] nodes joined)[{analytics-node}{xfFlmyVORRupiq6ip5MckA}{WY6_KU89RWmmZxM7JM3KHQ}{127.0.0.1}{127.0.0.1:9300}{dilm}{ml.machine_memory=8464900096, xpack.installed=true, ml.max_open_jobs=20} elect leader, BECOME_MASTER_TASK, FINISH_ELECTION], term: 8, version: 167, reason: master node changed {previous , current [{analytics-node}{xfFlmyVORRupiq6ip5MckA}{WY6_KU89RWmmZxM7JM3KHQ}{127.0.0.1}{127.0.0.1:9300}{dilm}{ml.machine_memory=8464900096, xpack.installed=true, ml.max_open_jobs=20}]}
[2019-12-12T09:36:49,822][INFO ][o.e.c.s.ClusterApplierService] [analytics-node] master node changed {previous , current [{analytics-node}{xfFlmyVORRupiq6ip5MckA}{WY6_KU89RWmmZxM7JM3KHQ}{127.0.0.1}{127.0.0.1:9300}{dilm}{ml.machine_memory=8464900096, xpack.installed=true, ml.max_open_jobs=20}]}, term: 8, version: 167, reason: Publication{term=8, version=167}
[2019-12-12T09:36:49,997][INFO ][o.e.h.AbstractHttpServerTransport] [analytics-node] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2019-12-12T09:36:50,004][INFO ][o.e.n.Node ] [analytics-node] started
[2019-12-12T09:36:50,333][INFO ][o.e.c.s.ClusterSettings ] [analytics-node] updating [cluster.remote.scarf-ec2.seeds] from [] to [["10.6.138.69:8081"]]
[2019-12-12T09:36:50,485][INFO ][o.e.c.s.ClusterSettings ] [analytics-node] updating [cluster.remote.scarf-ec2.seeds] from [] to [["10.6.138.69:8081"]]
[2019-12-12T09:36:50,535][INFO ][o.e.c.s.ClusterSettings ] [analytics-node] updating [cluster.remote.scarf-ec2.seeds] from [] to [["10.6.138.69:8081"]]
[2019-12-12T09:36:51,184][INFO ][o.e.l.LicenseService ] [analytics-node] license [444b452d-62f4-424c-81eb-f032c85971b1] mode [basic] - valid
[2019-12-12T09:36:51,187][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [analytics-node] Active license is now [BASIC]; Security is disabled
[2019-12-12T09:36:51,212][INFO ][o.e.g.GatewayService ] [analytics-node] recovered [6] indices into cluster_state
[2019-12-12T09:36:53,449][INFO ][o.e.c.r.a.AllocationService] [analytics-node] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana_1][0]]]).
[2019-12-12T09:37:11,666][WARN ][o.e.t.RemoteClusterConnection] [analytics-node] fetching nodes from external cluster [scarf-ec2] failed
org.elasticsearch.transport.ConnectTransportException: [10.6.138.69:8081] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:976) ~[elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$3(ActionListener.java:161) ~[elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.4.0.jar:7.4.0]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.4.0.jar:7.4.0]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:68) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:500) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:493) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:472) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:413) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:538) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:531) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:111) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:323) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:685) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:597) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:551) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: no further information: /10.6.138.69:8081
Caused by: java.net.ConnectException: Connection timed out: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:336) ~[?:?]
... 7 more
[2019-12-12T09:37:11,705][WARN ][o.e.t.RemoteClusterService] [analytics-node] failed to update seed list for cluster: scarf-ec2
org.elasticsearch.transport.ConnectTransportException: [10.6.138.69:8081] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:976) ~[elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$3(ActionListener.java:161) ~[elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.4.0.jar:7.4.0]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.4.0.jar:7.4.0]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:68) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:500) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:493) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:472) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:413) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:538) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:531) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:111) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:323) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:685) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:597) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:551) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: no further information: /10.6.138.69:8081
Caused by: java.net.ConnectException: Connection timed out: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:336) ~[?:?]

My cluster name is scarf-initializr and node is scarf-initializr-analytics.

Thanks.

This looks like a ES misconfiguration. The resolution is in the error message:
cdn.com/elastic/user_avatar/discuss.elastic.co/alejandro_bartolomeo/40/42054_2.png) Alejandro_Bartolomeo:

at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured

In this case, I think it's enough to set discovery.seed_hosts: [] .

Thanks
Rashmi

Thanks @rashmi

Will check this. But it worked for me without changing this when I started again after a day. Not sure what was the reason .

Great! Thanks for posting

Thanks
Rashmi

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.