How to start a elastic cluster 8.1 with three or more nodes

I have the first node, but I don't know what is next. What do I have to do to define the master and join new nodes to the cluster.
Is there a guide or something explained ?

Thanks in advanced.

The number of nodes and your configuration is determined by your needs. If you are just testing out you can run one node, but an enterprise cluster usually has 3 masters, data nodes, etc. You may need dedicated ingest nodes as well, but that all depends on how big it is and how much data you want to retain. For a smaller stack, you could just run 3 nodes which are both master and data together.

Thanks for the prompt answer.
In fact, I've been testing one node for a couple of months, but I will need to test also a cluster or multinode deployment. What I have seen is that there is no current info on how to create a cluster with the newer version. Last year I tested a cluster with version 7 and it was quite simple, but now since version 8.1 comes with security enable it is not clear to me. I would appreciate any help on this.

Ah ok. So it is specifically an issue with version 8.1 for you. That will help others diagnose your issue. I am personally still in 7.x myself, and only use it with Kubernetes.

Did you see Add and remove nodes in your cluster | Elasticsearch Guide [8.1] | Elastic

Yes, I have been following every document I find, but there are missing steps that I would appreciate having at hand for creating the cluster.
What I have made so far:

  1. Created the servers and I have installed Elasticsearch 8.1
  2. On first node I installed kibana
  3. I created the enrollment token for second node
  4. I reconfigured second node with the created enrollment token
  5. I don't know where to go from here.

Start the second node.
If it doesn't startup and join with the first node, please paste the relevant lines from the log file and we'll help diagnose what went wrong.

1 Like
cluster.name: SF-ELK-CLUSTER
node.name: SF-ELK-CL01
node.roles: [ data, master ]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 10.14.1.215
discovery.seed_hosts: ["10.14.1.215", "10.14.1.216"]
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["SF-ELK-CL01"]
http.host: [_local_, _site_]
transport.host: [_local_, _site_]


cluster.name: SF-ELK-CLUSTER
node.name: SF-ELK-CL02
node.roles: [ data ]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 10.14.1.216
cluster.initial_master_nodes: ["SF-ELK-CL01"]
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
discovery.seed_hosts: ["10.14.1.215:9300"]


[2022-05-03T15:35:41,009][WARN ][o.e.c.c.Coordinator      ] [SF-ELK-CL01] received join request from [{SF-ELK-CL02}{6RTcN3xlR0Kh9ef0dsvzoA}{DBFDzZvrQwyudO9x69tSpQ}{10.14.1.216}{10.14.1.216:9300}{d}{xpack.installed=true}] but could not connect back to the joining node
org.elasticsearch.transport.ConnectTransportException: [SF-ELK-CL02][10.14.1.216:9300] connect_exception
        at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1106) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:305) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.core.CompletableContext.lambda$addListener$0(CompletableContext.java:33) ~[elasticsearch-core-8.2.0.jar:8.2.0]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
        at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:887) ~[?:?]
        at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2325) ~[?:?]
        at org.elasticsearch.core.CompletableContext.addListener(CompletableContext.java:36) ~[elasticsearch-core-8.2.0.jar:8.2.0]
        at org.elasticsearch.transport.netty4.Netty4TcpChannel.addConnectListener(Netty4TcpChannel.java:141) ~[?:?]
        at org.elasticsearch.transport.TcpTransport.initiateConnection(TcpTransport.java:399) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:361) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.transport.ClusterConnectionManager.internalOpenConnection(ClusterConnectionManager.java:348) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.transport.ClusterConnectionManager.connectToNodeOrRetry(ClusterConnectionManager.java:199) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.transport.ClusterConnectionManager.connectToNode(ClusterConnectionManager.java:98) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:411) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:395) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:572) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$1(JoinHelper.java:112) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:253) ~[?:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$3.onResponse(SecurityServerTransportInterceptor.java:315) ~[?:?]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$3.onResponse(SecurityServerTransportInterceptor.java:305) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:614) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:255) ~[?:?]
        at org.elasticsearch.xpack.security.transport.ServerTransportFilter.lambda$inbound$1(ServerTransportFilter.java:113) ~[?:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:93) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:199) ~[?:?]
        at org.elasticsearch.xpack.security.transport.ServerTransportFilter.inbound(ServerTransportFilter.java:105) ~[?:?]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:326) ~[?:?]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:287) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.2.0.jar:8.2.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: No route to host: 10.14.1.216/10.14.1.216:9300
Caused by: java.net.NoRouteToHostException: No route to host
        at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
        at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:710) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        ... 1 more
[2022-05-03T15:35:42,010][WARN ][o.e.c.c.Coordinator      ] [SF-ELK-CL01] received join request from [{SF-ELK-CL02}{6RTcN3xlR0Kh9ef0dsvzoA}{DBFDzZvrQwyudO9x69tSpQ}{10.14.1.216}{10.14.1.216:9300}{d}{xpack.installed=true}] but could not connect back to the joining node
org.elasticsearch.transport.ConnectTransportException: [SF-ELK-CL02][10.14.1.216:9300] connect_exception
        at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1106) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:305) ~[elasticsearch-8.2.0.jar:8.2.0]
        at org.elasticsearch.core.CompletableContext.lambda$addListener$0(CompletableContext.java:33) ~[elasticsearch-core-8.2.0.jar:8.2.0]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
        at org.elasticsearch.core.CompletableContext.completeExceptionally(CompletableContext.java:48) ~[elasticsearch-core-8.2.0.jar:8.2.0]
        at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:63) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:710) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: No route to host: 10.14.1.216/10.14.1.216:9300
Caused by: java.net.NoRouteToHostException: No route to host
        at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
        at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]
        ... 7 more

I found the issue and I have corrected it. This was solved. It seems the node has joined the cluster.