Concept of Elasticsearch cluster

pyerunka · September 27, 2019, 9:56am

Hello,

I want to apply elasticsearch cluster on my environment. I have done some changes in elasticsearch.yaml file like:
cluster.name: Test
node.name: node-1
node.data: true
node.master: true

what values we have to set for:
I need short description about them, so that i can relate things.
network.host
discovery.seed_hosts
discovery.zen.ping.unicast.hosts

Regards,
Priyanka

gitreseteasy · September 27, 2019, 11:28am

I'm not sure if i'm qualified to give you any good analogies but I found some docs that may be helpful for you.

https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html

Dominic_Page · September 27, 2019, 11:58am

Hi @pyerunka

If you are running one node, congratulations, you already have an Elasticsearch cluster!

All of the settings you mention are useful when adding more nodes to the cluster, so you might want to check out this link, which explains how to do that

https://www.elastic.co/guide/en/elasticsearch/reference/current/add-elasticsearch-nodes.html

Hope this helps

pyerunka · September 30, 2019, 10:03am

Hello @Dominic_Page,

Thanks for your help!!!
I am able to create cluster with one master node on one of the server A. Now i have to create one data node in same cluster.
so my question is, in order to create 2nd node, do i have to do changes on another server B? that means do i have to install elasticsearch on another server B? and do the changes in elasticsearch.yml file where i will keep cluster name same as mentioned in settings file of server A?

Guide me.

Regards,
Priyanka

Dominic_Page · September 30, 2019, 2:10pm

Hi @pyerunka you will need to set up a few things.

Your master node will need to advertise it's availability and accept connections.

You can configure this with the setting network.host, see:

https://www.elastic.co/guide/en/elasticsearch/reference/7.3/modules-transport.html#modules-transport

(try setting this to _site_ to get started, assuming your new data node is running on a separate host)

Your new data node will need a few things:

cluster.name should match the master node cluster.name setting, so they know they belong together

node.name should differ from the master node node.name setting, as each node needs a unique identity

discovery.seed_hosts should include the server where the master node is running, so the new data node can find the master

As you are running with 2 nodes, I'd also recommend to set

node.master to false on the new node

You might want to change that again if you add a third node in the future, see: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html

pyerunka · October 1, 2019, 9:01am

Hello @Dominic_Page,
Thanks for your reply!!!
I have done some settings as mentioned by you in my config file i.e. elasticsearch.yml file.
please find below config for the same:
for Master node:

cluster.name: Cluster
node.name: Master-node
node.master: true
node.data: false
network.host: abcd.com
http.port: 9200
transport.tcp.port: 9300
discovery.seed_hosts: ["*.*.*.*:9300", "*.*.*.*:9300"]
cluster.initial_master_nodes: ["Master-node"]
bootstrap.memory_lock: true

for data node:

cluster.name: Cluster
node.name: Data-node1
node.master: false
node.data: true
network.host: abcd.com
discovery.seed_hosts: ["*.*.*.*:9300"]
http.port: 9200
transport.tcp.port:9300
bootstrap.memory_lock: true

I am getting an error while running an elasticsearch service:

Use 'elasticsearch-node re purpose' tool to clean up at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.2.0.jar:7.2.0]at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.2.0.jar:7.2.0] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.2.0.jar:7.2.0] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.2.0.jar:7.2.0]at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-c li-7.2.0.jar:7.2.0]at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115 ) ~[elasticsearch-7.2.0.jar:7.2.0]at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)~[elasticsearch-7.2.0.jar:7.2.0] Caused by: java.lang.IllegalStateException: Node is started with node.data=false but has shard data: [E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices-RMolVw6R6mB9S5pXiVCFg\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\021AEDKOQpWHbgu0mtJnYw\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\0aVGB4-USFCg6R2z9tNv2A\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\6cvvxh3uSruMh_6n1RHeqA\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\8S06taQPQxqlxaM_fC2o3Q\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\AJXnq5aDTg68Klk--3RosQ\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\blRwOmq8RDidmBWTBS3CJA\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\CNiHe7rPQuOFp-KRaypt6Q\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\CqVyVAyGRgykv6dHttOh6Q\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\dHCq7u4IQQyWHWw6-0V0hA\0, E:\ES\elasticsear ch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\FQNCZjlkT0eejLoJ2YqoJQ\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\FR4iNHI3RxWwMLVOnDYgYQ\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\ghu_VwK3QWiHnFZjO4gG5w\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\HKidfystSN2F5BDEgsCJWA\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data \nodes\0\indices\hkIRXWUGQjq8_E4K9nnfgA\0, E:\ES\elasticsearch-7.2.0-windows-x86 _64\elasticsearch-7.2.0\data\nodes\0\indices\Hz6cyMaxTtWuC3tGDykyQg\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\ih7YVGSpTheLyHNdiL0aWg\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\ikzHxg5OQsyA9oo7GVsULg\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\KHoIwNfmRPC2JhsNO4PYIQ\0, E:\ES \elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\L3lUgCM3QfmJM18C6jyYqA\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\LSzTjda0SLGZ0qdpJmYNsQ\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\Mdme-VFNRw2jteLw09J95A\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\mEGqIIvcRQyX7sPn4wBPRw\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\MJYYlMlNSSiud_qza0DgBQ\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\N781S4EFQqia7WCkAh5IuQ\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\o4ZNehjVTb2aJqFmbVjaCQ\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\qkx9YwVRShClcwmbyagqhw\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\sBt8-9g6Q6iJyQF8PsLM8A\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\SOSGCEDwRqGQOV0EEYKwaw\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\xeS1fg8VQTixeGOBuvsLIw\0, E:\ES\elasticsearch-7.2.0-windows-x86_64\elasticsearch-7.2.0\data\nodes\0\indices\xhND5y4_Q6ymi-k Cz0BHsQ\0]. Use 'elasticsearch-node repurpose' tool to clean up at org.elasticsearch.env.NodeEnvironment.ensureNoShardData(NodeEnvironment.java:1065) ~[elasticsearch-7.2.0.jar:7.2.0]at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:324 ) ~[elasticsearch-7.2.0.jar:7.2.0] at org.elasticsearch.node.Node.(Node.java:271) ~[elasticsearch-7.2.0.jar:7.2.0] at org.elasticsearch.node.Node.(Node.java:251) ~[elasticsearch-7.2.0.jar:7.2.0]at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.2.0.jar:7.2.0]at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159 ) ~[elasticsearch-7.2.0.jar:7.2.0]> ... 6 more

Kindly guide me on the same.

Regards,
Priyanka

Dominic_Page · October 1, 2019, 4:46pm

Hi @pyerunka there are a few issues here - let's take them step by step:

Caused by: java.lang.IllegalStateException: Node is started with node.data=false but has shard data

Elasticsearch is not happy because "Master-node" already has some data (in a "shard") and that data is no longer accessible because

node.data: false

It is normal in smaller clusters for nodes to have multiple roles, so I suggest setting this value to true.

network.host needs to identify an interface (not a host), or use one of the meta settings. Assuming your nodes are running on separate hosts, you might want to set that to _site_ for both nodes.
discovery.seed_hosts needs to specify one or more concrete hosts where e.g. the data node can find the master node to start their relationship. So, port 9300 is good, though as that is the default you don't need to specify. You will need to change *.*.*.* to an IP or hostname:

so for the data node, set that to the host of the master node
and for the master node, set that to the host of the data node

Hope this helps

pyerunka · October 3, 2019, 6:48am

Hello @Dominic_Page,

Thanks for reply!!
I have done changes as suggested. Now on master node ES is working. but on my data node it is throwing an error.
Please find below elasticsearch.yml file for data node followed by error:

cluster.name: Cluster
node.name: Data-node1
node.master: false
node.data: true
node.ingest: false
network.host:abcd.com
discovery.zen.ping.unicast.hosts: [abcd.com, xyz.com]
transport.tcp.port: 9300-9400
path.data: /elasticsearch/data/
path.logs: /var/log/elasticsearch/
path.repo: /elasticsearch/repo/
bootstrap.memory_lock: true
http.port: 9200
network.bind_host:   Ipaddres of data node

Error:

[2019-10-03T06:22:43,884][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [Data-node1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: BindHttpException[Failed to bind to [9200]]; nested: BindException[Address already in use: bind];
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwar
eCommand.java:86) ~[elasticsearch-7.2.0.jar:7.2.0]at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.2.0.jar:7.2.0]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.2.0.jar:7.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)~[elasticsearch-7.2.0.jar:7.2.0]
Caused by: org.elasticsearch.http.BindHttpException: Failed to bind to [9200] at org.elasticsearch.http.AbstractHttpServerTransport.bindAddress(AbstractHttpServerTransport.java:175) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.http.AbstractHttpServerTransport.bindServer(AbstractHttpServerTransport.java:142) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.http.netty4.Netty4HttpServerTransport.doStart(Netty4HttpServerTransport.java:228) ~[?:?]
at org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport.doStart(SecurityNetty4HttpServerTransport.java:63) ~[?:?]
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:59) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.node.Node.start(Node.java:724) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:273) ~[elasticsearch-7.2.0.jar:7.2.0]at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.2.0.jar:7.2.0]
... 6 more
Caused by: java.net.BindException: Address already in use: bind
        at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
        at sun.nio.ch.Net.bind(Net.java:433) ~[?:?]
        at sun.nio.ch.Net.bind(Net.java:425) ~[?:?]
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) ~[?:?]
        at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:132) ~[?:?]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:563) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1332) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:503) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:488) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:984) ~[?:?]
        at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:259) ~[?:?]
        at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366) ~[?:?]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:405) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201][2019-10-03T06:22:43,899][INFO ][o.e.n.Node               ] [Data-node1] stopping ...
[2019-10-03T06:22:43,930][INFO ][o.e.x.w.WatcherService   ] [Data-node1] stoppin
g watch service, reason [shutdown initiated]
[2019-10-03T06:22:43,993][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [Data-node1]
[controller/12492] [Main.cc@150] Ml controller exiting
[2019-10-03T06:22:43,993][INFO ][o.e.x.m.p.NativeController] [Data-node1] Nativecontroller process has stopped - no new native processes can be started
[2019-10-03T06:22:43,993][INFO ][o.e.n.Node               ] [Data-node1] stopped

[2019-10-03T06:22:43,993][INFO ][o.e.n.Node               ] [Data-node1] closing
 ...
[2019-10-03T06:22:44,009][INFO ][o.e.n.Node               ] [Data-node1] closed

Thanks & Regards,
Priyanka

MiTschMR · October 3, 2019, 8:33am

Hi @pyerunka

From the logs it is clearly stated that the port 9200 is already in use:

BindHttpException[Failed to bind to [9200]]; nested: BindException[Address already in use: bind];

Something on your system is blocking the port. You are not running another instance of Elasticsearch on this node? Please post both elasticsearch.yml, from the data and the master node.

pyerunka · October 3, 2019, 8:46am

Hello @MiTschMR,

Thanks for reply!!!
I am trying to run elasticsearch on data node, at that time i am getting this error.
and i do not see any PID using port 9200 in my task manager.
Please find below yml file for both master and data node:

Master Node:

cluster.name: Cluster
cluster.remote.connect: false
network.host: abcd.com
transport.host: xyz.com
transport.tcp.port: 9300-9400
node.name: Master-node
discovery.zen.minimum_master_nodes: 1
node.master: true
node.data: true
node.ingest: true
discovery.zen.ping.unicast.hosts: [abcd.com, xyz.com]
cluster.initial_master_nodes: ["Master-node"]
bootstrap.memory_lock: true
http.port: 9200
network.bind_host:  ip address of master node

Data Node:

cluster.name: Cluster
node.name: Data-node1
node.master: false
node.data: true
node.ingest: false
network.host: abcd.com
discovery.zen.ping.unicast.hosts: [abcd.com, xyz.com]
transport.tcp.port: 9300-9400
path.data: /elasticsearch/data/
path.logs: /var/log/elasticsearch/
path.repo: /elasticsearch/repo/
bootstrap.memory_lock: true
http.port: 9200
network.bind_host:   ip address of datanode

Thanks,
Priyanka

MiTschMR · October 3, 2019, 9:06am

Hi @pyerunka

Your problem might be related to the following setting: network.host
Please read common network settings.

You might need to change the network.host value on the Master Node to xyz.com and on the Data Node to abcd.com.

Hope this helps.

pyerunka · October 3, 2019, 9:37am

Hello @MiTschMR,

i tried using above solution. but still i am getting same error.

Regards,
Priyanka Yerunkar.

Dominic_Page · October 3, 2019, 12:30pm

Hi @pyerunka @MiTschMR

@pyerunka - are you running on 2 separate hosts?

Your config shows both nodes running on abcd.com and the master node is also running on xyz.com

@pyerunka Please can you clarify

What is the host name of the server where the master node is running?
Is the data node on the same server?
Otherwise, what is the host name of the server where the data node is running?

Thanks!

pyerunka · October 4, 2019, 3:21am

Hello @Dominic_Page,

Thanks for reply!!!!
Please find below answers:
Yes, I am running on 2 separate hosts.
The host name of the server where the master node is running is abcd.com
Data node is not on the same server.
The host name of the server where the data node is running is xyz.com

For network.host setting i have changed the host name.
For master node network.host is xyz.com and for data node it is abcd.com.

Regards,
Priyanka Yerunkar

Dominic_Page · October 4, 2019, 11:51am

Hi @pyerunka

For the Master Node, you will need to remove:

transport.host: xyz.com

for 2 reasons because:

network.host already specifies this and
it should reference the node itself, not a different node

also remove the configurations

cluster.remote.connect
discoery.zen.minimum_master_nodes // only for previous versions
network.bind_host // let network host do this

For the data node you might change

network.host: xyz.com

and again remove

network.bind_host // let network host do this

Hope this helps

pyerunka · October 7, 2019, 5:14am

Hello @Dominic_Page,
Thanks for reply!!
I did changes as suggested by you. But still i am not able start ES service on data node.
Please find below logs for the same:[2019-10-07T04:13:17,324][INFO ]

[2019-10-07T04:13:19,965][INFO ][o.e.n.Node               ] [Data-node1] initialized
[2019-10-07T04:13:19,965][INFO ][o.e.n.Node               ] [Data-node1] starting ...
[2019-10-07T04:13:20,168][INFO ][o.e.t.TransportService   ] [Data-node1] publish_address {136.252.130.86:9300}, bound_addresses {[::1]:9300},{136.252.130.86:9300}, {10.130.22.92:9300}
[2019-10-07T04:13:20,184][INFO ][o.e.b.BootstrapChecks    ] [Data-node1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-10-07T04:13:30,211][WARN ][o.e.c.c.ClusterFormationFailureHelper] [Data-node1] master not discovered yet: have discovered []; discovery will continue using [136.252.129.115:9300] from hosts providers and [{Data-node1}{vMrL87DfRfmZV46pYXb2HA}{G8joowAhTTCkxse_lRnEYQ}{nl0123epc0911.dir.slb.com}{136.252.130.86:9300}{ml.machine_memory=51539005440, xpack.installed=true, ml.max_open_jobs=20}] from
last-known cluster state; node term 0, last-accepted version 0 in term 0
[2019-10-07T04:13:40,233][WARN ][o.e.c.c.ClusterFormationFailureHelper] [Data-node1] master not discovered yet: have discovered []; discovery will continue using [136.252.129.115:9300] from hosts providers and [{Data-node1}{vMrL87DfRfmZV46pYXb2HA}{G8joowAhTTCkxse_lRnEYQ}{nl0123epc0911.dir.slb.com}{136.252.130.86:9300}{ml.machine_memory=51539005440, xpack.installed=true, ml.max_open_jobs=20}] from
last-known cluster state; node term 0, last-accepted version 0 in term 0
[2019-10-07T04:13:50,227][WARN ][o.e.n.Node               ] [Data-node1] timed out while waiting for initial discovery state - timeout: 30s
[2019-10-07T04:13:50,243][WARN ][o.e.c.c.ClusterFormationFailureHelper] [Data-node1] master not discovered yet: have discovered []; discovery will continue using [136.252.129.115:9300] from hosts providers and [{Data-node1}{vMrL87DfRfmZV46pYXb2HA}{G8joowAhTTCkxse_lRnEYQ}{nl0123epc0911.dir.slb.com}{136.252.130.86:9300}{ml.machine_memory=51539005440, xpack.installed=true, ml.max_open_jobs=20}] from
last-known cluster state; node term 0, last-accepted version 0 in term 0
[2019-10-07T04:13:50,290][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [Data-node1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: BindHttpException[Failed to bind to [9200]]; nested: BindException[Address already in use: bind];
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.2.0.jar:7.2.0]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.2.0.jar:7.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)~[elasticsearch-7.2.0.jar:7.2.0]
Caused by: org.elasticsearch.http.BindHttpException: Failed to bind to [9200]at org.elasticsearch.http.AbstractHttpServerTransport.bindAddress(AbstractHttpServerTransport.java:175) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.http.AbstractHttpServerTransport.bindServer(AbstractHttpServerTransport.java:142) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.http.netty4.Netty4HttpServerTransport.doStart(Netty4HttpServerTransport.java:228) ~[?:?]
at org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport.doStart(SecurityNetty4HttpServerTransport.java:63) ~[?:?]
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:59) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.node.Node.start(Node.java:724) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:273) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.2.0.jar:7.2.0]
... 6 more
Caused by: java.net.BindException: Address already in use: bind
        at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
        at sun.nio.ch.Net.bind(Net.java:433) ~[?:?]
        at sun.nio.ch.Net.bind(Net.java:425) ~[?:?]
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) ~[?:?] at
io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:132) ~[?:?]
at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:563) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1332) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:503) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:488) ~[?:?] at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:984) ~[?:?]
at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:259) ~[?:?]
at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366) ~[?:?]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[?:?]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:405) ~[?:?]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) ~[?:?]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) ~[?:?]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
    [2019-10-07T04:13:51,290][INFO ][o.e.n.Node               ] [Data-node1] stopping ...
    [2019-10-07T04:13:51,290][INFO ][o.e.x.w.WatcherService   ] [Data-node1] stopping watch service, reason [shutdown initiated]
    [2019-10-07T04:13:51,556][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [Data-node1]
    [controller/13144] [Main.cc@150] Ml controller exiting

Regards,
Priyanka

Dominic_Page · October 7, 2019, 9:23am

Hi @pyerunka

The error is Caused by: java.net.BindException: Address already in use: bind

Can you check that you do not already have an Elasticsearch Process running on you data node, for example by running:

ps -ef | grep org.elasticsearch.bootstrap.Elasticsearch

or the pslist equivalent if you are running Windows

If you already have a process running, you will need to stop that process so that the new process can run using your new configuration

pyerunka · October 7, 2019, 10:24am

Hello @Dominic_Page,

Thanks for reply!!
I have checked that, no ES process is running on data node. Also checked port no 9200, port is not used by another process. Now i am checking if firewall is blocking communication between two hosts.

Regards,
Priyanka

Dominic_Page · October 7, 2019, 11:15am

Hi @pyerunka

You can try running

curl -X GET abcd.com:9300

from your data node to see if you can contact the master node

If the response is This is not an HTTP port then you have made contact successfully

pyerunka · October 7, 2019, 11:35am

Hello @Dominic_Page,

I got below response when i tried to connect master node from data node:
curl: (7) Failed to connect to abcd.com port 9300: Timed out.

Regards,
Priyanka