Got error when starting a elastic server cluster

Elastic Stack Elasticsearch
1

I got error like below in both servers

[2018-11-11T12:38:21,155][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2018-11-11T12:38:21,157][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.
[2018-11-11T12:38:21,158][WARN ][o.e.b.JNANatives         ] Increase RLIMIT_MEMLOCK, soft limit: 16777216, hard limit: 16777216
[2018-11-11T12:38:21,158][WARN ][o.e.b.JNANatives         ] These can be adjusted by modifying /etc/security/limits.conf, for example: 
	# allow user 'elasticsearch' mlockall
	elasticsearch soft memlock unlimited
	elasticsearch hard memlock unlimited
[2018-11-11T12:38:21,158][WARN ][o.e.b.JNANatives         ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2018-11-11T12:38:21,304][INFO ][o.e.n.Node               ] [linux-1] initializing ...
[2018-11-11T12:38:21,401][INFO ][o.e.e.NodeEnvironment    ] [linux-1] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [24.6gb], net total_space [28.9gb], types [ext4]
[2018-11-11T12:38:21,401][INFO ][o.e.e.NodeEnvironment    ] [linux-1] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-11-11T12:38:21,402][INFO ][o.e.n.Node               ] [linux-1] node name [linux-1], node ID [h03oGLGESzqeHmeNJLl0LQ]
[2018-11-11T12:38:21,402][INFO ][o.e.n.Node               ] [linux-1] version[6.4.3], pid[11401], build[oss/deb/fe40335/2018-10-30T23:17:19.084789Z], OS[Linux/4.15.0-1030-azure/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_181/25.181-b13]
[2018-11-11T12:38:21,403][INFO ][o.e.n.Node               ] [linux-1] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true,  -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=oss, -Des.distribution.type=deb]
[2018-11-11T12:38:22,242][INFO ][o.e.p.PluginsService     ] [linux-1] loaded module [aggs-matrix-stats]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService     ] [linux-1] loaded module [analysis-common]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService     ] [linux-1] loaded module [lang-expression]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService     ] [linux-1] loaded module [rank-eval]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService     ] [linux-1] loaded module [reindex]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService     ] [linux-1] loaded module [repository-url]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService     ] [linux-1] loaded module [transport-netty4]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService     ] [linux-1] loaded module [tribe]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService     ] [linux-1] no plugins loaded
[2018-11-11T12:38:25,636][INFO ][o.e.d.DiscoveryModule    ] [linux-1] using discovery type [zen]
[2018-11-11T12:38:26,235][INFO ][o.e.n.Node               ] [linux-1] initialized
[2018-11-11T12:38:26,235][INFO ][o.e.n.Node               ] [linux-1] starting ...
[2018-11-11T12:38:26,561][ERROR][o.e.b.Bootstrap          ] [linux-1] Exception
org.elasticsearch.transport.BindTransportException: Failed to bind to [9093]
	at org.elasticsearch.transport.TcpTransport.bindToPort(TcpTransport.java:821) ~[elasticsearch-6.4.3.jar:6.4.3]
	at org.elasticsearch.transport.TcpTransport.bindServer(TcpTransport.java:786) ~[elasticsearch-6.4.3.jar:6.4.3]
	at org.elasticsearch.transport.netty4.Netty4Transport.doStart(Netty4Transport.java:134) ~[?:?]
	at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:66) 

Caused by: java.net.BindException: Cannot assign requested address
	at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
	at sun.nio.ch.Net.bind(Net.java:433) ~[?:?]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
[2018-11-11T12:38:26,581][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [linux-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: BindTransportException[Failed to bind to [9093]]; nested: BindException[Cannot assign requested address];
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.4.3.jar:6.4.3]

	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.3.jar:6.4.3]
Caused by: org.elasticsearch.transport.BindTransportException: Failed to bind to [9093]
	at org.elasticsearch.transport.TcpTransport.bindToPort(TcpTransport.java:821) ~[elasticsearch-6.4.3.jar:6.4.3]
	at org.elasticsearch.transport.TcpTransport.bindServer(TcpTransport.java:786) ~[elasticsearch-6.4.3.jar:6.4.3]

	... 6 more
Caused by: java.net.BindException: Cannot assign requested address
	at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
	at sun.nio.ch.Net.bind(Net.java:433) ~[?:?]
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) ~[?:?]

	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
[2018-11-11T12:38:27,284][INFO ][o.e.n.Node               ] [linux-1] stopping ...
[2018-11-11T12:38:27,288][INFO ][o.e.n.Node               ] [linux-1] stopped
[2018-11-11T12:38:27,288][INFO ][o.e.n.Node               ] [linux-1] closing ...
[2018-11-11T12:38:27,321][INFO ][o.e.n.Node               ] [linux-1] closed

My config file for servers as below

Linux-1 server

cluster.name: linux-elk
node.name: linux-1
node.data: true
node.ingest: true
node.master: false
node.max_local_storage_nodes: 2
indices.query.bool.max_clause_count: 10000
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 24.95.245.313
http.port: 9092
transport.tcp.port: 9093
discovery.zen.ping.unicast.hosts: 
     - 178.51.190.47
discovery.zen.minimum_master_nodes: 1

Linux-2 server

cluster.name: linux-elk
node.name: linux-2
node.data: true
node.ingest: false
node.master: true
node.max_local_storage_nodes: 2
indices.query.bool.max_clause_count: 10000
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 178.51.190.47
http.port: 9092
transport.tcp.port: 9093
discovery.zen.ping.unicast.hosts: 
     - 24.95.245.313
discovery.zen.minimum_master_nodes: 1

plese help ? what i need to do now ?

2

Hi,

Verify you do not have special chars in your network.host line.

And are you certain this line is valid :

bye
Xavier

3

it is sever ipv4 address. what do you mean by special chars?

How should I verify it?

4

The unicast.hosts line seems strange for me.
In my configuration I have all on the same line:

network.host: ["10.xxx.xxx.xxx", "172.yyy.yyy.yyy"]

Specials chars can be control chars you can't see and look like a space. It's just an idea.

5

As you have specified a non-default transport port, you need to include this in your unicast list.

6

I tried it and got same error again. But if i change network.host to 0.0.0.0 in both nodes and start nodes, both working well. That time i don;t have this error ?

7

please help??? anybody

8

hi,

can you past your elasticsearch.yml, the last log error and the result of ifconfig ?

thx

9

what is ifconfig?

10

hi, ifconfig is a linux command to retrieve all IPs of the server. In order to see if it's matching with your configuration.

bye,
Xavier

11

Can you please show your updated config?

12

Linux-1 server

cluster.name: linux-elk
node.name: linux-1
node.data: true
node.ingest: true
node.master: false
node.max_local_storage_nodes: 2
indices.query.bool.max_clause_count: 10000
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 24.95.245.313
http.port: 9092
transport.tcp.port: 9093
discovery.zen.ping.unicast.hosts: ["178.51.190.47:9093"]
discovery.zen.minimum_master_nodes: 1

Linux-2 server

cluster.name: linux-elk
node.name: linux-2
node.data: true
node.ingest: false
node.master: true
node.max_local_storage_nodes: 2
indices.query.bool.max_clause_count: 10000
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 178.51.190.47
http.port: 9092
transport.tcp.port: 9093
discovery.zen.ping.unicast.hosts: ["24.95.245.313:9093"]
discovery.zen.minimum_master_nodes: 1

This is my updated config file of both nodes. still giving same error

13

Log into each host and verify that you can talent successfully to post 9093 on the other host. The config looks Ok as far as I can see, so it is possible it is a network or firewall issue.

If these IP addressed are public, also make sure that you have secured your cluster properly so it is not openly accessible from the internet.

14

when i change

network.host to 0.0.0.0

in both nodes all working fine and cluster started with green status.
But when i give original ip it giving error. why is that ?

15

any help ?

16

Hi did you solve your problem ?

17

not yet

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.