Elasticsearch 5.2.0 unable to form cluster

Elastic Stack Elasticsearch
1

Hi Team,
I have difficulty setting up Elasticsearch cluster in version 5.2.0. This is the first time i am using Version 5.x. I have been happily using v2.4 so far.

Please find my elasticsearch.yml file below. I have the same settings on both the nodes (trying to build a 2 node cluster)

However, if i specify this setting, my node will start without any issues, but as a standalone node.

http.host: 10.1.1.10
http.port: 9200

i understand that this transport.host setting wont form a cluster, i even tried without this setting. it still doesnt work.

transport.host: localhost
transport.tcp.port: 9300

If i remove transport.host from the setting, i am getting bootstrap errors.

I have provided the full elasticsearch.yml (using the same settings on both the nodes)
and my log file....

cluster.name: elasticsearch
cluster.routing.allocation.awareness.force.cloud.values:  zone

node.name: Elkprod1
node.master: true
node.data: true
node.max_local_storage_nodes: 1

path.conf: /usr/local/etc/elasticsearch
path.data: /data/elasticsearch
path.logs: /usr/local/var/log/elasticsearch

network.host: 0.0.0.0
http.port: 9200

transport.host: localhost
transport.tcp.port: 9300

gateway.expected_nodes: 1

Log File...

[2017-03-01T17:05:09,618][INFO ][o.e.p.PluginsService     ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] no plugins loaded
[2017-03-01T17:05:11,358][INFO ][o.e.n.Node               ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] initialized
[2017-03-01T17:05:11,358][INFO ][o.e.n.Node               ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] starting ...
[2017-03-01T17:05:11,442][WARN ][i.n.u.i.MacAddressUtil   ] Failed to find a usable hardware address from the network interfaces; using random bytes: bd:e2:6b:bb:af:90:b9:e9
[2017-03-01T17:05:11,497][INFO ][o.e.t.TransportService   ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] publish_address {10.1.1.10:9300}, bound_addresses {0.0.0.0:9300}
[2017-03-01T17:05:11,501][INFO ][o.e.b.BootstrapChecks    ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-03-01T17:05:11,503][ERROR][o.e.b.Bootstrap          ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] node validation exception
bootstrap checks failed
max file descriptors [64000] for elasticsearch process is too low, increase to at least [65536]
system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2017-03-01T17:05:11,505][INFO ][o.e.n.Node               ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] stopping ...
[2017-03-01T17:05:11,583][INFO ][o.e.n.Node               ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] stopped
[2017-03-01T17:05:11,584][INFO ][o.e.n.Node               ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] closing ...
[2017-03-01T17:05:11,604][INFO ][o.e.n.Node               ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] closed
2
3

@dadoonet

  1. I have fixed the file descriptor problem
  2. The Linux kernel i am using does not have SecComp. So i have disabled system call filter checks.

bootstrap.system_call_filter

So, both the problems are fixed now. Now i am able to start my node without any bootstrap check errors. But still the cluster mode is not working as expected.

This is my config file now..

cluster.name: elasticsearch
cluster.routing.allocation.awareness.force.cloud.values: zone
node.name: elknode-1

node.master: true
node.data: true
node.max_local_storage_nodes: 1

path.conf: /usr/local/etc/elasticsearch
path.data: /data/elasticsearch
path.logs: /usr/local/var/log/elasticsearch


bootstrap.system_call_filter: false

http.host: 10.1.1.10
http.port: 9200


gateway.expected_nodes: 1

bootstrap.memory_lock: true

discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: ["10.1.1.10","10.1.1.11"]

Getting the below in the log file.. Doesnt seem to working in a master-client cluster

[2017-03-01T21:53:21,060][INFO ][o.e.n.Node ] [opselk-197323431-1-198091806.Elkprod.apps-ops.boitop.com] initializing ...
[2017-03-01T21:53:21,143][INFO ][o.e.e.NodeEnvironment ] [opselk-197323431-1-198091806.Elkprod.apps-ops.boitop.com] using [1] data paths, mounts [[/data (/dev/mapper/opselk-volume--197323431--1)]], net usable_space [700.5gb], net total_space [738.1gb], spins? [possibly], types [ext4]
[2017-03-01T21:53:21,144][INFO ][o.e.e.NodeEnvironment ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] heap size [11.4gb], compressed ordinary object pointers [true]
[2017-03-01T21:53:21,145][INFO ][o.e.n.Node ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] node name [opselk-197323431-1-198091806.Elkprod.apps-ops.boitop.cdcprod5.prod.walmart.com], node ID [IUd6bgP-TPmv8NFfwv1tcA]
[2017-03-01T21:53:21,146][INFO ][o.e.n.Node ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] version[5.2.0], pid[4538], build[24e05b9/2017-01-24T19:52:35.800Z], OS[Linux/2.6.32-642.el6.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_51/25.51-b03]
[2017-03-01T21:53:21,936][INFO ][o.e.p.PluginsService ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] loaded module [aggs-matrix-stats]
[2017-03-01T21:53:21,936][INFO ][o.e.p.PluginsService ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] loaded module [ingest-common]
[2017-03-01T21:53:21,936][INFO ][o.e.p.PluginsService ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] loaded module [lang-expression]
[2017-03-01T21:53:21,936][INFO ][o.e.p.PluginsService ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] loaded module [lang-groovy]
[2017-03-01T21:53:21,936][INFO ][o.e.p.PluginsService ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] loaded module [lang-mustache]
[2017-03-01T21:53:21,936][INFO ][o.e.p.PluginsService ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] loaded module [lang-painless]
[2017-03-01T21:53:21,936][INFO ][o.e.p.PluginsService ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] loaded module [percolator]
[2017-03-01T21:53:21,936][INFO ][o.e.p.PluginsService ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] loaded module [reindex]
[2017-03-01T21:53:21,936][INFO ][o.e.p.PluginsService ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] loaded module [transport-netty3]
[2017-03-01T21:53:21,936][INFO ][o.e.p.PluginsService ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] loaded module [transport-netty4]
[2017-03-01T21:53:21,937][INFO ][o.e.p.PluginsService ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] no plugins loaded
[2017-03-01T21:53:23,854][INFO ][o.e.n.Node ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] initialized
[2017-03-01T21:53:23,854][INFO ][o.e.n.Node ] [opselk-197323431-1-198091806.Elkprod.apps-ops.com] starting ...
[2017-03-01T21:53:23,940][WARN ][i.n.u.i.MacAddressUtil ] Failed to find a usable hardware address from the network interfaces; using random bytes: 34:be:ba:27:d4:93:a9:27
[2017-03-01T21:53:24,003][INFO ][o.e.t.TransportService ] [opselk-197323431-1-198091806.Elkprod.apps-ops.boitop.com] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2017-03-01T21:53:54,047][WARN ][o.e.n.Node ] [opselk-197323431-1-198091806.Elkprod.apps-ops.boitop.com] timed out while waiting for initial discovery state - timeout: 30s
[2017-03-01T21:53:54,057][INFO ][o.e.h.HttpServer ] [opselk-197323431-1-198091806.Elkprod.apps-ops.boitop.com] publish_address {10.1.1.10:9200}, bound_addresses {10.1.1.10:9200}
[2017-03-01T21:53:54,058][INFO ][o.e.n.Node ] [opselk-197323431-1-198091806.Elkprod.apps-ops.boitop.com] started

4

I'd probably change http.host by network.host

5

@dadoonet changing it to http.host to network.host worked !!! Thank You very much !

Is it a good idea to have both the nodes in the cluster as MASTER nodes?

6

Ideally you should have 3 nodes, see https://www.elastic.co/guide/en/elasticsearch/guide/2.x/important-configuration-changes.html#_minimum_master_nodes

7

Can you please help me to understand this better ?

When you say minimum three nodes ? 2 master and 1 client node ?

What is the downside if i have only 2 nodes in a cluster and both the nodes act master ? What is the problem if i dont have the third node.

If you have two regular nodes, you are in a conundrum. A quorum would be 2, but this means a loss of one node will make your cluster inoperable. A setting of 1 will allow your cluster to function, but doesn’t protect against split brain. It is best to have a minimum of three nodes in situations like this.

8

That's exactly what will happen.

9

Ok. Thank You. !

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.