Memory lock not working

Elastic Stack Elasticsearch
1

I'm running a Fedora 25 ec2 image with elasticsearch 5.1.1

This are my settings:

    /etc/elasticsearch/elasticsearch.yml
    bootstrap.memory_lock: true

    /usr/lib/systemd/system/elasticsearch.service
    LimitMEMLOCK=infinity

    /etc/sysconfig/elasticsearch
    MAX_LOCKED_MEMORY=unlimited

    /etc/security/limits.conf
    * soft memlock unlimited
    * hard memlock unlimited

even after that I'm still getting:

Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: unlimited
memory locking requested for elasticsearch process but memory is not locked
2

Hi @Mormaii,

How did you install Elasticsearch (.zip, .rpm) as the configuration procedure is a little bit different for each option (see docs)?

To me it seems some of the limits are not detected, as the error message says that the soft limit is 65536.

Daniel

3

I installed it via the rpm packages. I don't understand why it doesn't work on fedora. I used it on centos before and had no problem at all.

4

Hi @Mormaii,

I tried to reproduce your problem on a fresh install of Fedora but it worked fine for me. Here is what I did:

sudo dnf install java-1.8.0-openjdk.x86_64
sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

# add the following contents with sudo vim /etc/yum.repos.d/elasticsearch.repo

[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

sudo dnf install elasticsearch

# Now change all config files that you've mentioned in your initial post (not reproducing it here again for brevity)

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service

I checked the logs and no warnings showed up. Next I checked via the API:

curl http://localhost:9200/_nodes?filter_path=**.mlockall\&pretty

This produced:

{
  "nodes" : {
    "-Xe_hH7ZSC6fp6ZF7UkngQ" : {
      "process" : {
        "mlockall" : true
      }
    }
  }
}

So it looks like some setting is not picked up in your environment. Did you try to reboot the machine?

Daniel

2 Likes
5

That's really strange. I've tried rebooting the machine and setting the ulimit by hand on the current session too and it works to no avail. If I remove the mlockall it starts up just fine but when I try to enable it then elasticsearch fails.

6

Hi @Mormaii,

one more possibility is that your tmp directory is mounted with noexec so you can try set a different tmp directory for Elasticsearch by adding this to config/jvm.options:

-Djava.io.tmpdir=/path/to/temp/dir

Daniel

1 Like
7

I tried setting the the tmpdir to the same data/log dir that elasticsearch uses and I still get the same error

node validation exception bootstrap checks failed
memory locking requested for elasticsearch process but memory is not locked
8

Do you work over OpenSSH and /etc/ssh/sshd_config contains UsePAM no?

9

I work over OpenSSH on an EC2 instance. UsePAM is set to yes.

10

So https://bugzilla.redhat.com/show_bug.cgi?id=1308423 is not of any relevance, I guess.

11

Have you verified there are no conflicting configurations in any conf files in /etc/security/limits.d/?

12

There's no files on /etc/security/limits.d/

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.