Got error when starting a elastic server cluster


(Varun) #1

I got error like below in both servers

[2018-11-11T12:38:21,155][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2018-11-11T12:38:21,157][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.
[2018-11-11T12:38:21,158][WARN ][o.e.b.JNANatives         ] Increase RLIMIT_MEMLOCK, soft limit: 16777216, hard limit: 16777216
[2018-11-11T12:38:21,158][WARN ][o.e.b.JNANatives         ] These can be adjusted by modifying /etc/security/limits.conf, for example: 
	# allow user 'elasticsearch' mlockall
	elasticsearch soft memlock unlimited
	elasticsearch hard memlock unlimited
[2018-11-11T12:38:21,158][WARN ][o.e.b.JNANatives         ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2018-11-11T12:38:21,304][INFO ][o.e.n.Node               ] [linux-1] initializing ...
[2018-11-11T12:38:21,401][INFO ][o.e.e.NodeEnvironment    ] [linux-1] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [24.6gb], net total_space [28.9gb], types [ext4]
[2018-11-11T12:38:21,401][INFO ][o.e.e.NodeEnvironment    ] [linux-1] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-11-11T12:38:21,402][INFO ][o.e.n.Node               ] [linux-1] node name [linux-1], node ID [h03oGLGESzqeHmeNJLl0LQ]
[2018-11-11T12:38:21,402][INFO ][o.e.n.Node               ] [linux-1] version[6.4.3], pid[11401], build[oss/deb/fe40335/2018-10-30T23:17:19.084789Z], OS[Linux/4.15.0-1030-azure/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_181/25.181-b13]
[2018-11-11T12:38:21,403][INFO ][o.e.n.Node               ] [linux-1] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true,  -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=oss, -Des.distribution.type=deb]
[2018-11-11T12:38:22,242][INFO ][o.e.p.PluginsService     ] [linux-1] loaded module [aggs-matrix-stats]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService     ] [linux-1] loaded module [analysis-common]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService     ] [linux-1] loaded module [lang-expression]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService     ] [linux-1] loaded module [rank-eval]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService     ] [linux-1] loaded module [reindex]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService     ] [linux-1] loaded module [repository-url]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService     ] [linux-1] loaded module [transport-netty4]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService     ] [linux-1] loaded module [tribe]
[2018-11-11T12:38:22,243][INFO ][o.e.p.PluginsService     ] [linux-1] no plugins loaded
[2018-11-11T12:38:25,636][INFO ][o.e.d.DiscoveryModule    ] [linux-1] using discovery type [zen]
[2018-11-11T12:38:26,235][INFO ][o.e.n.Node               ] [linux-1] initialized
[2018-11-11T12:38:26,235][INFO ][o.e.n.Node               ] [linux-1] starting ...
[2018-11-11T12:38:26,561][ERROR][o.e.b.Bootstrap          ] [linux-1] Exception
org.elasticsearch.transport.BindTransportException: Failed to bind to [9093]
	at org.elasticsearch.transport.TcpTransport.bindToPort(TcpTransport.java:821) ~[elasticsearch-6.4.3.jar:6.4.3]
	at org.elasticsearch.transport.TcpTransport.bindServer(TcpTransport.java:786) ~[elasticsearch-6.4.3.jar:6.4.3]
	at org.elasticsearch.transport.netty4.Netty4Transport.doStart(Netty4Transport.java:134) ~[?:?]
	at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:66) 

Caused by: java.net.BindException: Cannot assign requested address
	at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
	at sun.nio.ch.Net.bind(Net.java:433) ~[?:?]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
[2018-11-11T12:38:26,581][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [linux-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: BindTransportException[Failed to bind to [9093]]; nested: BindException[Cannot assign requested address];
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.4.3.jar:6.4.3]

	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.3.jar:6.4.3]
Caused by: org.elasticsearch.transport.BindTransportException: Failed to bind to [9093]
	at org.elasticsearch.transport.TcpTransport.bindToPort(TcpTransport.java:821) ~[elasticsearch-6.4.3.jar:6.4.3]
	at org.elasticsearch.transport.TcpTransport.bindServer(TcpTransport.java:786) ~[elasticsearch-6.4.3.jar:6.4.3]

	... 6 more
Caused by: java.net.BindException: Cannot assign requested address
	at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
	at sun.nio.ch.Net.bind(Net.java:433) ~[?:?]
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) ~[?:?]

	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
[2018-11-11T12:38:27,284][INFO ][o.e.n.Node               ] [linux-1] stopping ...
[2018-11-11T12:38:27,288][INFO ][o.e.n.Node               ] [linux-1] stopped
[2018-11-11T12:38:27,288][INFO ][o.e.n.Node               ] [linux-1] closing ...
[2018-11-11T12:38:27,321][INFO ][o.e.n.Node               ] [linux-1] closed

My config file for servers as below

Linux-1 server

cluster.name: linux-elk
node.name: linux-1
node.data: true
node.ingest: true
node.master: false
node.max_local_storage_nodes: 2
indices.query.bool.max_clause_count: 10000
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 24.95.245.313
http.port: 9092
transport.tcp.port: 9093
discovery.zen.ping.unicast.hosts: 
     - 178.51.190.47
discovery.zen.minimum_master_nodes: 1

Linux-2 server

cluster.name: linux-elk
node.name: linux-2
node.data: true
node.ingest: false
node.master: true
node.max_local_storage_nodes: 2
indices.query.bool.max_clause_count: 10000
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 178.51.190.47
http.port: 9092
transport.tcp.port: 9093
discovery.zen.ping.unicast.hosts: 
     - 24.95.245.313
discovery.zen.minimum_master_nodes: 1

plese help ? what i need to do now ?


(Xavier Facq) #2

Hi,

Verify you do not have special chars in your network.host line.

And are you certain this line is valid :

bye
Xavier


(Varun) #3

it is sever ipv4 address. what do you mean by special chars?

How should I verify it?


(Xavier Facq) #4

The unicast.hosts line seems strange for me.
In my configuration I have all on the same line:

network.host: ["10.xxx.xxx.xxx", "172.yyy.yyy.yyy"]

Specials chars can be control chars you can't see and look like a space. It's just an idea.


(Christian Dahlqvist) #5

As you have specified a non-default transport port, you need to include this in your unicast list.


(Varun) #6

I tried it and got same error again. But if i change network.host to 0.0.0.0 in both nodes and start nodes, both working well. That time i don;t have this error ?


(Varun) #7

please help??? anybody


(Xavier Facq) #8

hi,

can you past your elasticsearch.yml, the last log error and the result of ifconfig ?

thx


(Varun) #9

what is ifconfig?


(Xavier Facq) #10

hi, ifconfig is a linux command to retrieve all IPs of the server. In order to see if it's matching with your configuration.

bye,
Xavier


(Christian Dahlqvist) #11

Can you please show your updated config?


(Varun) #12

Linux-1 server

cluster.name: linux-elk
node.name: linux-1
node.data: true
node.ingest: true
node.master: false
node.max_local_storage_nodes: 2
indices.query.bool.max_clause_count: 10000
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 24.95.245.313
http.port: 9092
transport.tcp.port: 9093
discovery.zen.ping.unicast.hosts: ["178.51.190.47:9093"]
discovery.zen.minimum_master_nodes: 1

Linux-2 server

cluster.name: linux-elk
node.name: linux-2
node.data: true
node.ingest: false
node.master: true
node.max_local_storage_nodes: 2
indices.query.bool.max_clause_count: 10000
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 178.51.190.47
http.port: 9092
transport.tcp.port: 9093
discovery.zen.ping.unicast.hosts: ["24.95.245.313:9093"]
discovery.zen.minimum_master_nodes: 1

This is my updated config file of both nodes. still giving same error


(Christian Dahlqvist) #13

Log into each host and verify that you can talent successfully to post 9093 on the other host. The config looks Ok as far as I can see, so it is possible it is a network or firewall issue.

If these IP addressed are public, also make sure that you have secured your cluster properly so it is not openly accessible from the internet.


(Varun) #14

when i change

network.host to 0.0.0.0

in both nodes all working fine and cluster started with green status.
But when i give original ip it giving error. why is that ?


(Varun) #15

any help ?