Systemd can not start elasticsearch

Elastic Stack Elasticsearch
1

We can start eleacticsearch 5.5.2 out side of systemd and it runs with out issue. But when we start eleacticsearch 5.5.2 with systemd it fails with the following error.
[1] bootstrap checks failed
[1]: max file descriptors [65535] for elasticsearch process is too low, increase to at least [65536]

we modified the following files to try to resolve the issue, see below:
/etc/security/limits.conf
/etc/security/limits.d/20-nproc.conf
/usr/lib/systemd/system/elasticsearch
These changes did not help. We must need a different configuration file changed. any id what that file is.

version of Linux
Linux 3.10.0-327.13.1.el7.x86_64 #1 SMP Mon Feb 29 13:22:02 EST 2016 x86_64 x86_64 x86_64 GNU/Linux

elasticsearch.service
[Unit]
Description=elasticsearch
After=network.target
Wants=network.target

[Service]
ExecStart=/opt/pki/elasticsearch/bin/elasticsearch -p /opt/pki/elasticsearch_pid/pid
Restart=on-failure
RestartSec=20
StartLimitInterval=20
StartLimitBurst=5
User=pkimd1m
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

/etc/security/limits.conf

  • soft nofile 65536
  • hard nofile 65536

/etc/security/limits.d/20-nproc.conf

  • soft nofile 65536
  • hard nofile 65536

/usr/lib/systemd/system/elasticsearch
LimitNOFILE=65536
LimitMEMLOCK=unlimited

2

My first question for you is why are you writing your own systemd unit file instead of using the RPM/Debian package which includes a tested systemd setup? It ships with a unit file that sets up the process to pass the OS-level bootstrap checks that we can control from the service, and we test that this works.

3

In particular, based on your kernel, you're on a RPM-based system that we support with our out-of-the-box systemd packaging.

4

My company has a standard way of packaging and deploying software. The elasticsearch tar file distributions fit into our home grown scheme where the RPM solution does not.

One of our requirements is to run the software as a non root user id.

Can you possible provide us with your systemd services file and what OS system files we need to change.

5

Elasticsearch does not run as root; in fact, we prevent running as root with a hard failure at runtime if a user tries to run as root.

As I mentioned, we provide these via the RPM. I understand that you're saying you can not use this, but the only reason given was not running as root which I explained above. So, I'd like to try again: please try our existing packaging, it's fully supported by us.

6

The reason we cannot use RPM has nothing to do with root.

My company has a standard way of packaging and deploying software. The elasticsearch tar file distributions fit into our home grown scheme where the RPM solution does not. We need to place the software in a very specific directory structure on the server.

7

Then it would have been better to have never mentioned it because it only adds confusion to the conversation, it's a distraction.

The tradeoff here is that we support and test our systemd unit files, not arbitrary unit files. This is something to keep in mind.

To your problem, I note that you have this:

Try changing that to LimitNOFILE=65536

The entries you have in /etc/security/limits.conf and /etc/security/limits.d do not make a difference, the limit on file descriptors for the process will come from the unit file when started by systemd.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.