How to start a elastic cluster 8.1 with three or more nodes

Adrian_Gtz · April 28, 2022, 4:09pm

I have the first node, but I don't know what is next. What do I have to do to define the master and join new nodes to the cluster.
Is there a guide or something explained ?

Thanks in advanced.

Damelon · April 28, 2022, 5:08pm

The number of nodes and your configuration is determined by your needs. If you are just testing out you can run one node, but an enterprise cluster usually has 3 masters, data nodes, etc. You may need dedicated ingest nodes as well, but that all depends on how big it is and how much data you want to retain. For a smaller stack, you could just run 3 nodes which are both master and data together.

Adrian_Gtz · April 28, 2022, 6:49pm

Thanks for the prompt answer.
In fact, I've been testing one node for a couple of months, but I will need to test also a cluster or multinode deployment. What I have seen is that there is no current info on how to create a cluster with the newer version. Last year I tested a cluster with version 7 and it was quite simple, but now since version 8.1 comes with security enable it is not clear to me. I would appreciate any help on this.

Damelon · April 28, 2022, 7:11pm

Ah ok. So it is specifically an issue with version 8.1 for you. That will help others diagnose your issue. I am personally still in 7.x myself, and only use it with Kubernetes.

warkolm · April 28, 2022, 11:33pm

Did you see Add and remove nodes in your cluster | Elasticsearch Guide [8.1] | Elastic

Adrian_Gtz · May 2, 2022, 7:36pm

Yes, I have been following every document I find, but there are missing steps that I would appreciate having at hand for creating the cluster.
What I have made so far:

Created the servers and I have installed Elasticsearch 8.1
On first node I installed kibana
I created the enrollment token for second node
I reconfigured second node with the created enrollment token
I don't know where to go from here.

TimV · May 3, 2022, 12:27am

Start the second node.
If it doesn't startup and join with the first node, please paste the relevant lines from the log file and we'll help diagnose what went wrong.

Adrian_Gtz · May 3, 2022, 7:49pm

cluster.name: SF-ELK-CLUSTER
node.name: SF-ELK-CL01
node.roles: [ data, master ]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 10.14.1.215
discovery.seed_hosts: ["10.14.1.215", "10.14.1.216"]
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["SF-ELK-CL01"]
http.host: [_local_, _site_]
transport.host: [_local_, _site_]


cluster.name: SF-ELK-CLUSTER
node.name: SF-ELK-CL02
node.roles: [ data ]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 10.14.1.216
cluster.initial_master_nodes: ["SF-ELK-CL01"]
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
discovery.seed_hosts: ["10.14.1.215:9300"]


[2022-05-03T15:35:41,009][WARN ][o.e.c.c.Coordinator      ] [SF-ELK-CL01] received join request from [{SF-ELK-CL02}{6RTcN3xlR0Kh9ef0dsvzoA}{DBFDzZvrQwyudO9x69tSpQ}{10.14.1.216}{10.14.1.216:9300}{d}{xpack.installed=true}] but could not connect back to the joining node
org.elasticsearch.transport.ConnectTransportException: [SF-ELK-CL02][10.14.1.216:9300] connect_exception
        at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1106) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:305) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.core.CompletableContext.lambda$addListener$0(CompletableContext.java:33) ~[elasticsearch-core-8.2.0.jar:8.2.0]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
        at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:887) ~[?:?]
        at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2325) ~[?:?]
        at org.elasticsearch.core.CompletableContext.addListener(CompletableContext.java:36) ~[elasticsearch-core-8.2.0.jar:8.2.0]
        at org.elasticsearch.transport.netty4.Netty4TcpChannel.addConnectListener(Netty4TcpChannel.java:141) ~[?:?]
        at org.elasticsearch.transport.TcpTransport.initiateConnection(TcpTransport.java:399) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:361) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.transport.ClusterConnectionManager.internalOpenConnection(ClusterConnectionManager.java:348) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.transport.ClusterConnectionManager.connectToNodeOrRetry(ClusterConnectionManager.java:199) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.transport.ClusterConnectionManager.connectToNode(ClusterConnectionManager.java:98) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:411) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:395) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:572) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$1(JoinHelper.java:112) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:253) ~[?:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$3.onResponse(SecurityServerTransportInterceptor.java:315) ~[?:?]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$3.onResponse(SecurityServerTransportInterceptor.java:305) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:614) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:255) ~[?:?]
        at org.elasticsearch.xpack.security.transport.ServerTransportFilter.lambda$inbound$1(ServerTransportFilter.java:113) ~[?:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:93) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:199) ~[?:?]
        at org.elasticsearch.xpack.security.transport.ServerTransportFilter.inbound(ServerTransportFilter.java:105) ~[?:?]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:326) ~[?:?]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:287) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.2.0.jar:8.2.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: No route to host: 10.14.1.216/10.14.1.216:9300
Caused by: java.net.NoRouteToHostException: No route to host
        at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
        at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:710) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        ... 1 more
[2022-05-03T15:35:42,010][WARN ][o.e.c.c.Coordinator      ] [SF-ELK-CL01] received join request from [{SF-ELK-CL02}{6RTcN3xlR0Kh9ef0dsvzoA}{DBFDzZvrQwyudO9x69tSpQ}{10.14.1.216}{10.14.1.216:9300}{d}{xpack.installed=true}] but could not connect back to the joining node
org.elasticsearch.transport.ConnectTransportException: [SF-ELK-CL02][10.14.1.216:9300] connect_exception
        at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1106) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:305) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.core.CompletableContext.lambda$addListener$0(CompletableContext.java:33) ~[elasticsearch-core-8.2.0.jar:8.2.0]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
        at org.elasticsearch.core.CompletableContext.completeExceptionally(CompletableContext.java:48) ~[elasticsearch-core-8.2.0.jar:8.2.0]
        at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:63) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:710) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: No route to host: 10.14.1.216/10.14.1.216:9300
Caused by: java.net.NoRouteToHostException: No route to host
        at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
        at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]
        ... 7 more

Adrian_Gtz · May 3, 2022, 8:10pm

I found the issue and I have corrected it. This was solved. It seems the node has joined the cluster.

system · May 31, 2022, 8:11pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.