Nodes not joining cluster on Centos 6.2 using ElasticSearch 5.2.2

Elastic Stack Elasticsearch
1

I have two nodes in different physical machines :

cluster.name: ss
node.name: node3
path.data: /elasticsearch/elasticsearch_data
path.logs: /elasticsearch/elasticsearch_logs
bootstrap.memory_lock: true
network.host: [local]
discovery.zen.ping.unicast.hosts: ["30.30.30.30"]

cluster.name: ss
node.name: node4
path.data: /elasticsearch/elasticsearch_data
path.logs: /elasticsearch/elasticsearch_logs
bootstrap.memory_lock: true
network.host: [local]
discovery.zen.ping.unicast.hosts: ["30.30.30.30"]

Both the machines are able to ping each other.

When i try the same from my local window boxes , they join the cluster. But, on the remote centos they are not joining.

If I change the network.host to the IP of the machine , elastic search service is not starting.

Please help. !!!

2

You need to share some logs.

3

Thanks @dadoonet .

Should i provide the ss.log which is generated for the node4 ?

Are you looking for something specific ?

Thanks
Abhishek.

4

This is log i get on the node4

[2017-03-30T09:46:18,077][INFO ][o.e.n.Node ] [node4] node name [node4], node ID [B0QFrXCWQEiu0Ap1FiTgtg]
[2017-03-30T09:46:18,079][INFO ][o.e.n.Node ] [node4] version[5.2.2], pid[62836], build[f9d9b74/2017-02-24T17:26:45.835Z], OS[Linux/2.6.32-504.el6.x86_64/amd64], JVM[.. 64-Bit Server VM/1.8.0_31/25.31-b07]
[2017-03-30T09:46:19,115][INFO ][o.e.p.PluginsService ] [node4] loaded module [aggs-matrix-stats]
[2017-03-30T09:46:19,115][INFO ][o.e.p.PluginsService ] [node4] loaded module [ingest-common]
[2017-03-30T09:46:19,115][INFO ][o.e.p.PluginsService ] [node4] loaded module [lang-expression]
[2017-03-30T09:46:19,115][INFO ][o.e.p.PluginsService ] [node4] loaded module [lang-groovy]
[2017-03-30T09:46:19,115][INFO ][o.e.p.PluginsService ] [node4] loaded module [lang-mustache]
[2017-03-30T09:46:19,115][INFO ][o.e.p.PluginsService ] [node4] loaded module [lang-painless]
[2017-03-30T09:46:19,115][INFO ][o.e.p.PluginsService ] [node4] loaded module [percolator]
[2017-03-30T09:46:19,115][INFO ][o.e.p.PluginsService ] [node4] loaded module [reindex]
[2017-03-30T09:46:19,115][INFO ][o.e.p.PluginsService ] [node4] loaded module [transport-netty3]
[2017-03-30T09:46:19,115][INFO ][o.e.p.PluginsService ] [node4] loaded module [transport-netty4]
[2017-03-30T09:46:19,116][INFO ][o.e.p.PluginsService ] [node4] no plugins loaded
[2017-03-30T09:46:21,528][INFO ][o.e.n.Node ] [node4] initialized
[2017-03-30T09:46:21,528][INFO ][o.e.n.Node ] [node4] starting ...
[2017-03-30T09:46:21,905][INFO ][o.e.t.TransportService ] [node4] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2017-03-30T09:46:21,914][WARN ][o.e.b.BootstrapChecks ] [node4] initial heap size [17179869184] not equal to maximum heap size [34359738368]; this can cause resize pauses and prevents mlockall from locking the entire heap
[2017-03-30T09:46:21,914][WARN ][o.e.b.BootstrapChecks ] [node4] memory locking requested for elasticsearch process but memory is not locked
[2017-03-30T09:46:21,915][WARN ][o.e.b.BootstrapChecks ] [node4] max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]
[2017-03-30T09:46:21,915][WARN ][o.e.b.BootstrapChecks ] [node4] system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2017-03-30T09:46:25,098][INFO ][o.e.c.s.ClusterService ] [node4] new_master {node4}{B0QFrXCWQEiu0Ap1FiTgtg}{Tq0PHwTtRjmrtTJbk94XeQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-03-30T09:46:25,153][INFO ][o.e.g.GatewayService ] [node4] recovered [0] indices into cluster_state
[2017-03-30T09:46:25,218][INFO ][o.e.h.HttpServer ] [node4] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2017-03-30T09:46:25,218][INFO ][o.e.n.Node ] [node4] started
[2017-03-30T09:46:26,588][INFO ][o.e.m.j.JvmGcMonitorService] [node4] [gc][5] overhead, spent [267ms] collecting in the last [1s

5

Your server only listen to 127.0.0.1 and is not accessible from outside.

You need to change network.host.

If I change the network.host to the IP of the machine , Elasticsearch service is not starting.

Yes. This is expected as I can see that:

[2017-03-30T09:46:21,914][WARN ][o.e.b.BootstrapChecks ] [node4] initial heap size [17179869184] not equal to maximum heap size [34359738368]; this can cause resize pauses and prevents mlockall from locking the entire heap
[2017-03-30T09:46:21,914][WARN ][o.e.b.BootstrapChecks ] [node4] memory locking requested for elasticsearch process but memory is not locked
[2017-03-30T09:46:21,915][WARN ][o.e.b.BootstrapChecks ] [node4] max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]
[2017-03-30T09:46:21,915][WARN ][o.e.b.BootstrapChecks ] [node4] system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk

When in production mode, you can't start anymore unless you fix those messages.

Please read Bootstrap Checks | Elasticsearch Guide [8.11] | Elastic

6

Thanks @dadoonet for the pointers. I will try to change the configurations as per the documents .

Also wanted to know , how i am able to connect across two different windows laptops with a very simialar configuration ?

Thanks
Abhishek,

7

Not sure I understand the question.

8

Thanks @dadoonet for the pointers.

I have done changes in the memory lock , threads and heap changes .

But now i am getting :

[2017-04-05T12:11:48,294][DEBUG][o.e.t.n.Netty4Transport ] [node-a]Bound profile [default] to address {10.33.A.B:9300}
[2017-04-05T12:11:48,296][INFO ][o.e.t.TransportService ] [node-a]publish_address {10.33.A.B:9300}, bound_addresses {10.33.A.B:9300}
[2017-04-05T12:11:48,302][INFO ][o.e.b.BootstrapChecks ] [node-a]bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-04-05T12:11:48,304][ERROR][o.e.b.Bootstrap ] [node-a]node validation exception
bootstrap checks failed
system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2017-04-05T12:11:48,307][INFO ][o.e.n.Node ] [node-a]stopping ...
[2017-04-05T12:11:48,595][INFO ][o.e.n.Node ] [node-a]stopped
[2017-04-05T12:11:48,595][INFO ][o.e.n.Node ] [node-a]closing ...
[2017-04-05T12:11:48,604][TRACE][o.e.n.Node ] [node-a]Close times for each service:
StopWatch 'node_close': running time = 0s/

if i make : transport.host: 10.33.A.B it is failing with above error

But, i make it as localhost , the loopback address its starting but obviously wont join the cluster

I am unable to get now what is the system call filter error that i should look for ?

Thanks
Abhishek.

9

There are logs higher up that tell you why the system call filters failed to install. I'll tell you that on CentOS 6, the kernel support for system call filters is not there. Therefore, you should explicitly disable the system call filters: bootstrap.system_call_filter=false.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.