Multiple Elasticsearch Nodes on One Host

seth.yes · April 18, 2017, 8:10pm

On ES 5.3, Ubuntu 14.04 I have a few servers with both HDD and SSD drives mapped to two different EXT4 filesystems:
/elasticsearch/warm # HDD /elasticsearch/hot # SSD

In order to increase my indexing capacity, I want to start two different ES nodes on each of these servers.

I couldn't really find much helpful documentation on doing this so I gave it a shot with what I know about ES but I've been running into issues. I navigated to the ES home (/usr/share/elasticsearch) and ran the command

 sudo -u elasticsearch ./bin/elasticsearch

The primary instance of ES (using the HDD) has configuration at /etc/elasticsearch
This secondary instance ES (Using the SSD) has configuration at /usr/share/elasticsearch/config

Primary elasticsearch.yml:

 cluster.name: elastic-cluster1

 node.name: warm-node

# DATA NODE ROLE
 node.master: false
 node.data: true
 node.ingest: false
 path.data: /elasticsearch/warm/data/ #HDD
 path.logs: /elasticsearch/warm/logs/
 bootstrap.memory_lock: true
 network.host: 10.191.5.54
 network.bind_host: 0.0.0.0
 http.port: 9200
 discovery.zen.ping.unicast.hosts: ["legitimate-hostnames-in-my-cluster"]
 discovery.zen.minimum_master_nodes: 3
 gateway.recover_after_nodes: 5
 xpack.security.enabled: false

Secondary elasticsearch.yml:

 cluster.name: elastic-cluster1
 node.name: hot-node
# DATA NODE ROLE
 node.master: false
 node.data: true
 node.ingest: false
 path.data: /elasticsearch/hot/data/ #SSD
 path.logs: /elasticsearch/warm/logs/hot/ #hot logs to HDD
 bootstrap.memory_lock: true
 network.bind_host: 0.0.0.0
 http.port: 9202
 transport.tcp.port: 9302
 discovery.zen.ping.unicast.hosts: ["legitimate-hostnames-in-my-cluster"]
 discovery.zen.minimum_master_nodes: 3
 gateway.recover_after_nodes: 5
 xpack.security.enabled: false

The first node running on the HDD is working fine, however when I attempt to startup the secondary node by issuing the command:

sudo -u elasticsearch ./bin/elasticsearch -Epath.conf=/usr/share/elasticsearch/config/

I receive the error:

[2017-04-18T14:09:12,482][INFO ][o.e.n.Node               ] [hot-node] initializing ...
[2017-04-18T14:09:12,576][INFO ][o.e.e.NodeEnvironment    ] [hot-node] using [1] data paths, mounts [[/elasticsearch/hot (/dev/mapper/elasticindex-elastichot)]], net usable_space [492.4gb], net total_space [518.9gb], spins? [no], types [ext4]
[2017-04-18T14:09:12,577][INFO ][o.e.e.NodeEnvironment    ] [hot-node] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-04-18T14:09:12,578][INFO ][o.e.n.Node               ] [hot-node] node name [hot-node], node ID [x-ghbSl5Rf6ObX50KRY4DQ]
[2017-04-18T14:09:12,579][INFO ][o.e.n.Node               ] [hot-node] version[5.3.0], pid[81161], build[3adb13b/2017-03-23T03:31:50.652Z], OS[Linux/4.4.0-57-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_121/25.121-b13]
[2017-04-18T14:09:13,988][INFO ][o.e.p.PluginsService     ] [hot-node] loaded module [aggs-matrix-stats]
[2017-04-18T14:09:13,989][INFO ][o.e.p.PluginsService     ] [hot-node] loaded module [ingest-common]
[2017-04-18T14:09:13,989][INFO ][o.e.p.PluginsService     ] [hot-node] loaded module [lang-expression]
[2017-04-18T14:09:13,989][INFO ][o.e.p.PluginsService     ] [hot-node] loaded module [lang-groovy]
[2017-04-18T14:09:13,989][INFO ][o.e.p.PluginsService     ] [hot-node] loaded module [lang-mustache]
[2017-04-18T14:09:13,989][INFO ][o.e.p.PluginsService     ] [hot-node] loaded module [lang-painless]
[2017-04-18T14:09:13,989][INFO ][o.e.p.PluginsService     ] [hot-node] loaded module [percolator]
[2017-04-18T14:09:13,990][INFO ][o.e.p.PluginsService     ] [hot-node] loaded module [reindex]
[2017-04-18T14:09:13,990][INFO ][o.e.p.PluginsService     ] [hot-node] loaded module [transport-netty3]
[2017-04-18T14:09:13,990][INFO ][o.e.p.PluginsService     ] [hot-node] loaded module [transport-netty4]
[2017-04-18T14:09:13,990][INFO ][o.e.p.PluginsService     ] [hot-node] loaded plugin [x-pack]
[2017-04-18T14:09:16,486][INFO ][o.e.n.Node               ] [hot-node] initialized
[2017-04-18T14:09:16,486][INFO ][o.e.n.Node               ] [hot-node] starting ...
[2017-04-18T14:09:16,805][INFO ][o.e.t.TransportService   ] [hot-node] publish_address {127.0.0.1:9302}, bound_addresses {[::]:9302}
[2017-04-18T14:09:16,816][INFO ][o.e.b.BootstrapChecks    ] [hot-node] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
ERROR: bootstrap checks failed
max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2017-04-18T14:09:16,826][INFO ][o.e.n.Node               ] [hot-node] stopping ...
[2017-04-18T14:09:16,875][INFO ][o.e.n.Node               ] [hot-node] stopped
[2017-04-18T14:09:16,875][INFO ][o.e.n.Node               ] [hot-node] closing ...
[2017-04-18T14:09:16,945][INFO ][o.e.n.Node               ] [hot-node] closed

So I can see it's using the proper elasticsearch.yml (due to filepaths and ports attempted to be opened), however none of the limits data is being utilized.

I've modified /usr/lib/systemd/system/elasticsearch.service to include:
LimitNOFILE=65536 LimitMEMLOCK=infinity
and /etc/default/elasticsearch to include:

MAX_OPEN_FILES=65536
MAX_LOCKED_MEMORY=unlimited

and /etc/security/limits.conf:

    elasticsearch soft memlock unlimited
    elasticsearch hard memlock unlimited
    elasticsearch  -  nofile  65536

and my jvm.options:

-Xms2g
-Xmx2g

of which there should be enough free memory:

             total       used       free     shared    buffers     cached
Mem:           94G        85G       9.0G       2.1M       223M        75G

However I'm still seeing the aforementioned error on startup:

ERROR: bootstrap checks failed
max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

Does anyone know what I'm doing wrong or can you point me to a good resource for starting two nodes on one server?

seth.yes · April 18, 2017, 9:03pm

I figured out how to get it running. I followed a post found here (excerpt below):

The actual way to raise your descriptors consists of editing three files:

/etc/security/limits.conf needs to have these lines in it:

soft nofile 64000

hard nofile 64000
root soft nofile 64000
root hard nofile 64000
The asterisk at the beginning of the first two lines means 'apply this rule to all users except root’, and you can probably guess that the last two lines set the limit only for the root user. The number at the end is of course, the new limit you’re setting. 64000 is a pretty safe number to use

/etc/pam.d/common-session needs to have this line in it:

session required pam_limits.so
/etc/pam.d/common-session-noninteractive also needs to have this line in it:

session required pam_limits.so

As well as adding to /etc/security/limits.conf:

    elasticsearch hard nofile 65536
    elasticsearch soft nofile 65536

Then restart the node.

system · May 16, 2017, 9:11pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.