Creating two node cluster in elasticsearch

Elastic Stack Elasticsearch
1

I have two instances of gcp and both have elasticsearch 7.3 installed on it master node also have kibana 7.3 installed on the same machine, i have set same cluster name for both the nodes declare one as master and one as data node, both node are running fine but after this curl -XGET -u elastic '10.128.0.26:9200/_cluster/health' i am getting this

{"cluster_name":"ElasticsearchStaging","status":"yellow","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":6,"active_shards":6,"relocating
_shards":0,"initializing_shards":0,"unassigned_shards":2,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_mil
lis":0,"active_shards_percent_as_number":75.0}

no.of node : 1

node-1 elasticsearech.yml file

cluster.name: ElasticsearchStaging
#xpack.security.enabled: true
#xpack.security.audit.enabled: true
#xpack.security.http.ssl.enabled: true
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
node.master: true
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch/
network.host: 10.128.0.26
#
# Set a custom port for HTTP:
#
http.port: 9200
discovery.seed_hosts:
   - 10.128.0.26:9200
   - 10.128.0.24:9200
cluster.initial_master_nodes:
   - node-1
   - node-2
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
xpack.security.audit.enabled: true

kibana.yml

server.port: 5601
server.host: 10.128.0.26
server.name: testing
elasticsearch.hosts: [http://10.128.0.26:9200](http://10.128.0.26:9200/
kibana.index: .kibana
elasticsearch.username: "kibana"
elasticsearch.password: "kibana"
logging.dest: /etc/kibana/kibana.log

node-1 elasticsearch.log

Aug 22 08:01:36 elastic-stage-vyakar elasticsearch[6177]:         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [
netty-transport-4.1.36.Final.jar:4.1.36.Final]
Aug 22 08:01:36 elastic-stage-vyakar elasticsearch[6177]:         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [
netty-transport-4.1.36.Final.jar:4.1.36.Final]
Aug 22 08:01:36 elastic-stage-vyakar elasticsearch[6177]:         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [ne
tty-transport-4.1.36.Final.jar:4.1.36.Final]
Aug 22 08:01:36 elastic-stage-vyakar elasticsearch[6177]:         at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-t
ransport-4.1.36.Final.jar:4.1.36.Final]
Aug 22 08:01:36 elastic-stage-vyakar elasticsearch[6177]:         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [
netty-transport-4.1.36.Final.jar:4.1.36.Final]
Aug 22 08:01:36 elastic-stage-vyakar elasticsearch[6177]:         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [
netty-transport-4.1.36.Final.jar:4.1.36.Final]
Aug 22 08:01:36 elastic-stage-vyakar elasticsearch[6177]:         at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-
4.1.36.Final.jar:4.1.36.Final]
Aug 22 08:01:36 elastic-stage-vyakar elasticsearch[6177]:         at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-tra
nsport-4.1.36.Final.jar:4.1.36.Final]
Aug 22 08:01:36 elastic-stage-vyakar elasticsearch[6177]:         at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.36.Final.
jar:4.1.36.Final]
Aug 22 08:01:36 elastic-stage-vyakar elasticsearch[6177]:         at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) [netty-transport-4.1.36.
Final.jar:4.1.36.Final]
Aug 22 08:01:36 elastic-stage-vyakar elasticsearch[6177]:         at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) [netty-transport-4.1.36.Final
.jar:4.1.36.Final]
Aug 22 08:01:36 elastic-stage-vyakar elasticsearch[6177]:         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.36.Final.jar:4.1.36.Fina
l]
Aug 22 08:01:36 elastic-stage-vyakar elasticsearch[6177]:         at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common
-4.1.36.Final.jar:4.1.36.Final]
Aug 22 08:01:36 elastic-stage-vyakar elasticsearch[6177]:         at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.36.Final.jar:4
.1.36.Final]
Aug 22 08:01:36 elastic-stage-vyakar elasticsearch[6177]:         at java.lang.Thread.run(Thread.java:835) [?:?]
Aug 22 08:01:36 elastic-stage-vyakar elasticsearch[6177]: [2019-08-22T08:01:36,429][INFO ][o.e.x.s.a.AuthenticationService] [node-1] Authentication of [kibana] was terminated 
by realm [reserved] - failed to authenticate user [kibana]
Aug 22 08:01:37 elastic-stage-vyakar elasticsearch[6177]: [2019-08-22T08:01:37,589][INFO ][o.e.c.r.a.AllocationService] [node-1] Cluster health status changed from [RED] to [Y
ELLOW] (reason: [shards started [[kibanaindx_1][0]] ...]).
Aug 22 08:01:38 elastic-stage-vyakar elasticsearch[6177]: [2019-08-22T08:01:38,632][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.management-beats] f
or index patterns [.management-beats]
2

node-2 elasticsearch.yml

cluster.name: ElasticsearchStaging
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-2
node.data: true
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
network.host: 10.128.0.24
#
# Set a custom port for HTTP:
#
http.port: 9200
discovery.seed_hosts:
   - 10.128.0.26:9200
   - 10.128.0.24:9200
cluster.initial_master_nodes:
   - node-1
   - node-2
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

node-2 elasticsearch.log

Aug 22 08:01:36 vyakar-stage-elastic-2 elasticsearch[16920]:         at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.36.Fin
al.jar:4.1.36.Final]
Aug 22 08:01:36 vyakar-stage-elastic-2 elasticsearch[16920]:         at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) [netty-transport-4.1.
36.Final.jar:4.1.36.Final]
Aug 22 08:01:36 vyakar-stage-elastic-2 elasticsearch[16920]:         at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) [netty-transport-4.1.36.Fi
nal.jar:4.1.36.Final]
Aug 22 08:01:36 vyakar-stage-elastic-2 elasticsearch[16920]:         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.36.Final.jar:4.1.36.F
inal]
Aug 22 08:01:36 vyakar-stage-elastic-2 elasticsearch[16920]:         at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-com
mon-4.1.36.Final.jar:4.1.36.Final]
Aug 22 08:01:36 vyakar-stage-elastic-2 elasticsearch[16920]:         at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.36.Final.ja
r:4.1.36.Final]
Aug 22 08:01:36 vyakar-stage-elastic-2 elasticsearch[16920]:         at java.lang.Thread.run(Thread.java:835) [?:?]
Aug 22 08:01:36 vyakar-stage-elastic-2 elasticsearch[16920]: Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 485454502f312e3020343030204261642052
6571756573740d0a636f6e74656e742d747970653a206170706c69636174696f6e2f6a736f6e3b20636861727365743d5554462d380d0a636f6e74656e742d6c656e6774683a203431390d0a0d0a7b226572726f72223a7
b22726f6f745f6361757365223a5b7b2274797065223a22696c6c6567616c5f617267756d656e745f657863657074696f6e222c22726561736f6e223a22696e76616c69642076657273696f6e20666f726d61743a20c386
c383c399c5b83fc299c286c2a1c2b42d5c7530303045c29fce9cc2954e4844c2877cc2bfc38ac38dc38e5360c2bec5b8c39bc39ac289c2afc2b75c7530303030285c75303031335c75303030325c75303031335c7530303
031c3802cc3802bc38030c3802fc38024c38023c38028c38027c380227d5d2c2274797065223a22696c6c6567616c5f617267756d656e745f657863657074696f6e222c22726561736f6e223a22696e76616c6964207665
7273696f6e20666f726d61743a20c386c383c399c5b83fc299c286c2a1c2b42d5c7530303045c29fce9cc2954e4844c2877cc2bfc38ac38dc38e5360c2bec5b8c39bc39ac289c2afc2b75c7530303030285c75303031335
c75303030325c75303031335c7530303031c3802cc3802bc38030c3802fc38024c38023c38028c38027c380227d2c22737461747573223a3430307d
Aug 22 08:01:36 vyakar-stage-elastic-2 elasticsearch[16920]:         at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1206) ~[netty-handler-4.1.36.Final.
jar:4.1.36.Final]
Aug 22 08:01:36 vyakar-stage-elastic-2 elasticsearch[16920]:         at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.36.Final.jar:4.1.36.Fi
nal]
Aug 22 08:01:36 vyakar-stage-elastic-2 elasticsearch[16920]:         at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:50
2) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final]
Aug 22 08:01:36 vyakar-stage-elastic-2 elasticsearch[16920]:         at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1
.36.Final.jar:4.1.36.Final]
Aug 22 08:01:36 vyakar-stage-elastic-2 elasticsearch[16920]:         ... 16 more
Aug 22 08:01:39 vyakar-stage-elastic-2 elasticsearch[16920]: [2019-08-22T08:01:39,298][INFO ][o.e.l.LicenseService     ] [node-2] license [93e41d31-fc5c-4bda-b4e0-47913389b1a1
] mode [basic] - valid
Aug 22 08:01:39 vyakar-stage-elastic-2 elasticsearch[16920]: [2019-08-22T08:01:39,302][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [node-2] Active license is now [BASIC]; S
ecurity is enabled
Aug 22 08:01:39 vyakar-stage-elastic-2 elasticsearch[16920]: [2019-08-22T08:01:39,349][INFO ][o.e.g.GatewayService     ] [node-2] recovered [1] indices into cluster_state
Aug 22 08:01:40 vyakar-stage-elastic-2 elasticsearch[16920]: [2019-08-22T08:01:40,653][INFO ][o.e.c.r.a.AllocationService] [node-2] Cluster health status changed from [RED] to
 [GREEN] (reason: [shards started [[.security-7][0]] ...]).
3

This error in the logfile on node-2 indicates you haven't properly configured your cluster for Transport Layer Security (TLS), which means the nodes can't communicate.

If you think your problem is to make the two nodes form a cluster I would start with turning off xpack.security (on both nodes) to see if they manage to form a cluster. Only if that works would I activate xpack.security again and try to solve that problem (see Tips to secure Elasticsearch clusters for free).

Good luck!

1 Like
4

Hello, now i comment down all the xpack.security thing in both the yml file but still number_of_node: 1

node-1 es.log

Aug 22 11:04:16 elastic-stage-vyakar elasticsearch[9842]:         at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:30
5) [elasticsearch-7.3.0.jar:7.3.0]
Aug 22 11:04:16 elastic-stage-vyakar elasticsearch[9842]:         at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:
139) [elasticsearch-7.3.0.jar:7.3.0]
Aug 22 11:04:16 elastic-stage-vyakar elasticsearch[9842]:         at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:264) 
[elasticsearch-7.3.0.jar:7.3.0]
Aug 22 11:04:16 elastic-stage-vyakar elasticsearch[9842]:         at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsea
rch-7.3.0.jar:7.3.0]
Aug 22 11:04:16 elastic-stage-vyakar elasticsearch[9842]:         at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:25
1) [elasticsearch-7.3.0.jar:7.3.0]
Aug 22 11:04:16 elastic-stage-vyakar elasticsearch[9842]:         at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:172) [elasticsearch-7.3
.0.jar:7.3.0]
Aug 22 11:04:16 elastic-stage-vyakar elasticsearch[9842]:         at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3
.0.jar:7.3.0]
Aug 22 11:04:16 elastic-stage-vyakar elasticsearch[9842]:         at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.3.0.j
ar:7.3.0]
Aug 22 11:04:16 elastic-stage-vyakar elasticsearch[9842]:         at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadConte
xt.java:758) [elasticsearch-7.3.0.jar:7.3.0]
Aug 22 11:04:16 elastic-stage-vyakar elasticsearch[9842]:         at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3
.0.jar:7.3.0]
Aug 22 11:04:16 elastic-stage-vyakar elasticsearch[9842]:         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
Aug 22 11:04:16 elastic-stage-vyakar elasticsearch[9842]:         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
Aug 22 11:04:16 elastic-stage-vyakar elasticsearch[9842]:         at java.lang.Thread.run(Thread.java:835) [?:?]
Aug 22 11:04:16 elastic-stage-vyakar elasticsearch[9842]: [2019-08-22T11:04:16,402][INFO ][o.e.c.r.a.AllocationService] [node-1] Cluster health status changed from [RED] to [Y
ELLOW] (reason: [shards started [[kibanaindx_1][0]] ...]).

node-2 es.log

Aug 22 11:55:43 vyakar-stage-elastic-2 elasticsearch[20535]: [2019-08-22T11:55:43,128][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-2] [controller/20623] [Main.cc@110] contr
oller (64 bit): Version 7.3.0 (Build ff2f774f78ce63) Copyright (c) 2019 Elasticsearch BV
Aug 22 11:55:44 vyakar-stage-elastic-2 elasticsearch[20535]: [2019-08-22T11:55:44,189][DEBUG][o.e.a.ActionModule       ] [node-2] Using REST wrapper from plugin org.elasticsea
rch.xpack.security.Security
Aug 22 11:55:45 vyakar-stage-elastic-2 elasticsearch[20535]: [2019-08-22T11:55:45,098][INFO ][o.e.d.DiscoveryModule    ] [node-2] using discovery type [zen] and seed hosts pro
viders [settings]
Aug 22 11:55:47 vyakar-stage-elastic-2 elasticsearch[20535]: [2019-08-22T11:55:47,220][INFO ][o.e.n.Node               ] [node-2] initialized
Aug 22 11:55:47 vyakar-stage-elastic-2 elasticsearch[20535]: [2019-08-22T11:55:47,229][INFO ][o.e.n.Node               ] [node-2] starting ...
Aug 22 11:55:47 vyakar-stage-elastic-2 elasticsearch[20535]: [2019-08-22T11:55:47,592][INFO ][o.e.t.TransportService   ] [node-2] publish_address {10.128.0.24:9300}, bound_add
resses {10.128.0.24:9300}
Aug 22 11:55:47 vyakar-stage-elastic-2 elasticsearch[20535]: [2019-08-22T11:55:47,615][INFO ][o.e.b.BootstrapChecks    ] [node-2] bound or publishing to a non-loopback address
, enforcing bootstrap checks
Aug 22 11:55:47 vyakar-stage-elastic-2 elasticsearch[20535]: [2019-08-22T11:55:47,681][INFO ][o.e.c.c.Coordinator      ] [node-2] cluster UUID [79SFZO4ERnGhRXvbDrliYw]
Aug 22 11:55:47 vyakar-stage-elastic-2 elasticsearch[20535]: [2019-08-22T11:55:47,944][INFO ][o.e.c.s.MasterService    ] [node-2] elected-as-master ([1] nodes joined)[{node-2}
{ZTi3ffcLQ1ms3KT6I2UOBA}{R8XmJ9_mTFGivgSOwknheA}{10.128.0.24}{10.128.0.24:9300}{dim}{ml.machine_memory=3872587776, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BE
COME_MASTER_TASK_, _FINISH_ELECTION_], term: 28, version: 128, reason: master node changed {previous [], current [{node-2}{ZTi3ffcLQ1ms3KT6I2UOBA}{R8XmJ9_mTFGivgSOwknheA}{10.1
28.0.24}{10.128.0.24:9300}{dim}{ml.machine_memory=3872587776, xpack.installed=true, ml.max_open_jobs=20}]}
Aug 22 11:55:48 vyakar-stage-elastic-2 elasticsearch[20535]: [2019-08-22T11:55:48,096][INFO ][o.e.c.s.ClusterApplierService] [node-2] master node changed {previous [], current
 [{node-2}{ZTi3ffcLQ1ms3KT6I2UOBA}{R8XmJ9_mTFGivgSOwknheA}{10.128.0.24}{10.128.0.24:9300}{dim}{ml.machine_memory=3872587776, xpack.installed=true, ml.max_open_jobs=20}]}, term
: 28, version: 128, reason: Publication{term=28, version=128}
Aug 22 11:55:48 vyakar-stage-elastic-2 elasticsearch[20535]: [2019-08-22T11:55:48,220][INFO ][o.e.h.AbstractHttpServerTransport] [node-2] publish_address {10.128.0.24:9200}, b
ound_addresses {10.128.0.24:9200}
Aug 22 11:55:48 vyakar-stage-elastic-2 elasticsearch[20535]: [2019-08-22T11:55:48,222][INFO ][o.e.n.Node               ] [node-2] started
Aug 22 11:55:48 vyakar-stage-elastic-2 elasticsearch[20535]: [2019-08-22T11:55:48,835][INFO ][o.e.l.LicenseService     ] [node-2] license [93e41d31-fc5c-4bda-b4e0-47913389b1a1
] mode [basic] - valid
Aug 22 11:55:48 vyakar-stage-elastic-2 elasticsearch[20535]: [2019-08-22T11:55:48,837][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [node-2] Active license is now [BASIC]; S
ecurity is disabled
Aug 22 11:55:48 vyakar-stage-elastic-2 elasticsearch[20535]: [2019-08-22T11:55:48,867][INFO ][o.e.g.GatewayService     ] [node-2] recovered [1] indices into cluster_state
Aug 22 11:55:49 vyakar-stage-elastic-2 elasticsearch[20535]: [2019-08-22T11:55:49,775][INFO ][o.e.c.r.a.AllocationService] [node-2] Cluster health status changed from [RED] to
 [GREEN] (reason: [shards started [[.security-7][0]] ...]).

NOTE:- for node-1 i set node.master: true for node-2 i set node.master: true and also node.data: true

5

That's better, now we have some useful information about the cluster formation.

From what you have revealed in the node logs it looks like both nodes form their own clusters, it's not obvious from the node-1 log but since it claims to have entered YELLOW state it must be in a cluster (else it would be RED). And since the node-2 log only lists itself, both as master and member of the cluster, it seems pretty clear that you've got two clusters with one node in each, rather than one cluster with two nodes. You can verify this by running

curl -XGET 'http://localhost:9200/_cat/nodes?pretty'

on both node-1 and node-2.

So, why doesn't the two nodes find each other? Well, there must be something wrong in the elasticsearch.yml configuration and one thing that sticks out is the port number you've specified in discovery.seed_hosts:

The discovery.seed_hosts is a new setting in ES7 so I haven't used it myself, but in the official documentation for Important discovery and cluster formation settings it says:

Each value should be in the form of host:port or host (where port defaults to the setting transport.profiles.default.port falling back to transport.port if not set).

So, the port must be the TCP port, not the HTTP. Since your http.port is 9200 you can't use this in the discovery.seed_hosts configuration. You will either have to use the TCP port (usually 9300) or just leave the port number out, since the default should work. In other words, you should change the setting to:

discovery.seed_hosts:
  - 10.128.0.26
  - 10.128.0.24

I hope that solves the problem :slight_smile:

6

Oh, by the way this has no effect:

By default both node.master and node.data are true (see the Node documentation) so your nodes will by default both be data nodes and master eligible.

7

Thanks for replying, but unfortunately even after doing this i am still getting single node detail by running this curl -XGET 'http://10.128.0.26:9200/_cat/nodes?pretty', localhost is not working for me, i also tried this pattern in both yml file but again not working10.128.0.26:9300, i also tried to change network.host: 0.0.0.0 but again didn't work out

node-1 es.log

Aug 22 13:02:15 elastic-stage-vyakar elasticsearch[12416]:         at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.36.Final
.jar:4.1.36.Final]
Aug 22 13:02:15 elastic-stage-vyakar elasticsearch[12416]:         at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) [netty-transport-4.1.36
.Final.jar:4.1.36.Final]
Aug 22 13:02:15 elastic-stage-vyakar elasticsearch[12416]:         at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) [netty-transport-4.1.36.Fina
l.jar:4.1.36.Final]
Aug 22 13:02:15 elastic-stage-vyakar elasticsearch[12416]:         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.36.Final.jar:4.1.36.Fin
al]
Aug 22 13:02:15 elastic-stage-vyakar elasticsearch[12416]:         at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-commo
n-4.1.36.Final.jar:4.1.36.Final]
Aug 22 13:02:15 elastic-stage-vyakar elasticsearch[12416]:         at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.36.Final.jar:
4.1.36.Final]
Aug 22 13:02:15 elastic-stage-vyakar elasticsearch[12416]:         at java.lang.Thread.run(Thread.java:835) [?:?]
Aug 22 13:02:16 elastic-stage-vyakar elasticsearch[12416]: [2019-08-22T13:02:16,245][INFO ][o.e.l.LicenseService     ] [node-1] license [87580efd-703f-4521-8736-377cba95a277] 
mode [basic] - valid
Aug 22 13:02:16 elastic-stage-vyakar elasticsearch[12416]: [2019-08-22T13:02:16,248][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [node-1] Active license is now [BASIC]; Sec
urity is disabled
Aug 22 13:02:16 elastic-stage-vyakar elasticsearch[12416]: [2019-08-22T13:02:16,280][INFO ][o.e.g.GatewayService     ] [node-1] recovered [5] indices into cluster_state
Aug 22 13:02:16 elastic-stage-vyakar elasticsearch[12416]: [2019-08-22T13:02:16,645][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [.management-beats] 
for index patterns [.management-beats]
Aug 22 13:02:18 elastic-stage-vyakar elasticsearch[12416]: [2019-08-22T13:02:18,884][INFO ][o.e.c.r.a.AllocationService] [node-1] Cluster health status changed from [RED] to [
YELLOW] (reason: [shards started [[kibanaindx_1][0]] ...]).

node-2 es.log

Aug 22 13:01:39 vyakar-stage-elastic-2 elasticsearch[22002]: [2019-08-22T13:01:39,603][INFO ][o.e.x.s.a.s.FileRolesStore] [node-2] parsed [0] roles from file [/etc/elasticsear
ch/roles.yml]
Aug 22 13:01:41 vyakar-stage-elastic-2 elasticsearch[22002]: [2019-08-22T13:01:41,333][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-2] [controller/22092] [Main.cc@110] contr
oller (64 bit): Version 7.3.0 (Build ff2f774f78ce63) Copyright (c) 2019 Elasticsearch BV
Aug 22 13:01:42 vyakar-stage-elastic-2 elasticsearch[22002]: [2019-08-22T13:01:42,283][DEBUG][o.e.a.ActionModule       ] [node-2] Using REST wrapper from plugin org.elasticsea
rch.xpack.security.Security
Aug 22 13:01:43 vyakar-stage-elastic-2 elasticsearch[22002]: [2019-08-22T13:01:43,229][INFO ][o.e.d.DiscoveryModule    ] [node-2] using discovery type [zen] and seed hosts pro
viders [settings]
Aug 22 13:01:45 vyakar-stage-elastic-2 elasticsearch[22002]: [2019-08-22T13:01:45,279][INFO ][o.e.n.Node               ] [node-2] initialized
Aug 22 13:01:45 vyakar-stage-elastic-2 elasticsearch[22002]: [2019-08-22T13:01:45,288][INFO ][o.e.n.Node               ] [node-2] starting ...
Aug 22 13:01:45 vyakar-stage-elastic-2 elasticsearch[22002]: [2019-08-22T13:01:45,624][INFO ][o.e.t.TransportService   ] [node-2] publish_address {10.128.0.24:9300}, bound_add
resses {10.128.0.24:9300}
Aug 22 13:01:45 vyakar-stage-elastic-2 elasticsearch[22002]: [2019-08-22T13:01:45,645][INFO ][o.e.b.BootstrapChecks    ] [node-2] bound or publishing to a non-loopback address
, enforcing bootstrap checks
Aug 22 13:01:45 vyakar-stage-elastic-2 elasticsearch[22002]: [2019-08-22T13:01:45,708][INFO ][o.e.c.c.Coordinator      ] [node-2] cluster UUID [79SFZO4ERnGhRXvbDrliYw]
Aug 22 13:01:45 vyakar-stage-elastic-2 elasticsearch[22002]: [2019-08-22T13:01:45,951][INFO ][o.e.c.s.MasterService    ] [node-2] elected-as-master ([1] nodes joined)[{node-2}
{ZTi3ffcLQ1ms3KT6I2UOBA}{fIwrSCz7SqyrOP1fv0t1vQ}{10.128.0.24}{10.128.0.24:9300}{dim}{ml.machine_memory=3872587776, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BE
COME_MASTER_TASK_, _FINISH_ELECTION_], term: 33, version: 153, reason: master node changed {previous [], current [{node-2}{ZTi3ffcLQ1ms3KT6I2UOBA}{fIwrSCz7SqyrOP1fv0t1vQ}{10.1
28.0.24}{10.128.0.24:9300}{dim}{ml.machine_memory=3872587776, xpack.installed=true, ml.max_open_jobs=20}]}
Aug 22 13:01:46 vyakar-stage-elastic-2 elasticsearch[22002]: [2019-08-22T13:01:46,169][INFO ][o.e.c.s.ClusterApplierService] [node-2] master node changed {previous [], current
 [{node-2}{ZTi3ffcLQ1ms3KT6I2UOBA}{fIwrSCz7SqyrOP1fv0t1vQ}{10.128.0.24}{10.128.0.24:9300}{dim}{ml.machine_memory=3872587776, xpack.installed=true, ml.max_open_jobs=20}]}, term
: 33, version: 153, reason: Publication{term=33, version=153}
Aug 22 13:01:46 vyakar-stage-elastic-2 elasticsearch[22002]: [2019-08-22T13:01:46,286][INFO ][o.e.h.AbstractHttpServerTransport] [node-2] publish_address {10.128.0.24:9200}, b
ound_addresses {10.128.0.24:9200}
Aug 22 13:01:46 vyakar-stage-elastic-2 elasticsearch[22002]: [2019-08-22T13:01:46,290][INFO ][o.e.n.Node               ] [node-2] started
Aug 22 13:01:46 vyakar-stage-elastic-2 elasticsearch[22002]: [2019-08-22T13:01:46,902][INFO ][o.e.l.LicenseService     ] [node-2] license [93e41d31-fc5c-4bda-b4e0-47913389b1a1
] mode [basic] - valid
Aug 22 13:01:46 vyakar-stage-elastic-2 elasticsearch[22002]: [2019-08-22T13:01:46,905][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [node-2] Active license is now [BASIC]; S
ecurity is disabled
Aug 22 13:01:46 vyakar-stage-elastic-2 elasticsearch[22002]: [2019-08-22T13:01:46,934][INFO ][o.e.g.GatewayService     ] [node-2] recovered [1] indices into cluster_state
Aug 22 13:01:47 vyakar-stage-elastic-2 elasticsearch[22002]: [2019-08-22T13:01:47,807][INFO ][o.e.c.r.a.AllocationService] [node-2] Cluster health status changed from [RED] to
 [GREEN] (reason: [shards started [[.security-7][0]] ...]).
8

I'm sorry that didn't work.

It looks like node-1 is stuck in its own cluster. Perhaps it has stored the old cluster state and is confused by that? By the way, what did the node-1 log say when it started up, before the error messages? Did it try to broadcast its IP like node-2 did?

If you haven't indexed much data yet, perhaps you could stop the nodes and delete all files in the data directories, e.g. in /var/lib/elasticsearch/, before starting up the nodes again? Then we know they can't remember the old cluster formation and should attempt to discover each other as a fresh cluster. Of course, I'm not sure if this is the solution but it could be worth a try.

Other than that I'm afraid I know too little about ES7 to be of more help, I'm still using ES6 and that version has a different master discovery mechanism without seed_hosts and initial_master_nodes.

9

Please don't be sorry i am very grateful that you are replying. By the way There is only one file in /var/lib/elasticsearch i.e nodes should i delete it and are you sure it will not effect elasticsearch of node-1 ?
Also do you think i should worry about this line in the log file of node-1

Caused by: org.elasticsearch.transport.RemoteTransportException: [node-1][10.128.0.26:9300][indices:data/read/get[s]]
Aug 23 05:41:10 elastic-stage-vyakar elasticsearch[3903]: Caused by: org.elasticsearch.index.shard.IllegalIndexShardStateException: CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED]
10

Everything that Elasticsearch needs to persist to disk, from its index data to cluster state, is stored under this directory so if the cluster gets corrupt (for instance because of the dreaded split-brain problem) the only way to solve it is often to erase all of the node's "memory".

In your case, since node-1 hasn't been able to join the cluster the data isn't valid and may conflict with data indexed in node-2, so it should be safe to stop the node and remove the nodes directory, which will be recreated and filled with data again when you start up the node.

As I wrote earlier, I am not sure if this will help you as I have no experience with troubleshooting the new ES 7. There could be other reasons why your 7.3 nodes won't form a cluster. But it could be worth a try to clear the memory of node-1 and start it up to see if it now joins the cluster. If it still fails there most be something else, either in the configuration or perhaps with a firewall setting that hinders node-1 from contacting node-2 to join the cluster.

This error seems to come because some process tried to acess a shard while it's recovering. This is typical when you restart a node, all shards must be recovered and only when they are in state STARTED can they be used for indexing new data. So I think you can disregard this error, your problem is not on the shard level but that node-1 doesn't join the cluster of node-2.

11

REALLY REALLY REALLY REALLY thanks brother it worked.

1 Like
12

I'm so happy it worked for you! Well done!

13

Sorry to disturb you again, but don't you think its a problem because every time i add a new node i have to delete my old data (i.e /var/lib/elasticsearch/nodes), as recently i added new node initially i won't work but after deleting old data of node-1 and node-2 it join the cluster.
Can you think of some alternative solution for this problem.

Thanks.

14

No, you shouldn't have to delete the old data directory again.

The only reason that was necessary was because the two nodes you configured had formed separate clusters when they started up the first time. But that shouldn't happen again, if you add a new data node and it is correctly configured it should connect to the cluster.

If you add a third node and it doesn't connect to the cluster there must be something wrong with its configuration, but I have no experience with 7.3 configuration yet so I don't know what could cause this.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.