Memory lock not working


#1

I'm running a Fedora 25 ec2 image with elasticsearch 5.1.1

This are my settings:

    /etc/elasticsearch/elasticsearch.yml
    bootstrap.memory_lock: true

    /usr/lib/systemd/system/elasticsearch.service
    LimitMEMLOCK=infinity

    /etc/sysconfig/elasticsearch
    MAX_LOCKED_MEMORY=unlimited

    /etc/security/limits.conf
    * soft memlock unlimited
    * hard memlock unlimited

even after that I'm still getting:

Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: unlimited
memory locking requested for elasticsearch process but memory is not locked

ElasticSearch Does Not Start on Clean Install
Can someone help me to add comment on following post
(Daniel Mitterdorfer) #2

Hi @Mormaii,

How did you install Elasticsearch (.zip, .rpm) as the configuration procedure is a little bit different for each option (see docs)?

To me it seems some of the limits are not detected, as the error message says that the soft limit is 65536.

Daniel


#3

I installed it via the rpm packages. I don't understand why it doesn't work on fedora. I used it on centos before and had no problem at all.


(Daniel Mitterdorfer) #4

Hi @Mormaii,

I tried to reproduce your problem on a fresh install of Fedora but it worked fine for me. Here is what I did:

sudo dnf install java-1.8.0-openjdk.x86_64
sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

# add the following contents with sudo vim /etc/yum.repos.d/elasticsearch.repo

[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

sudo dnf install elasticsearch

# Now change all config files that you've mentioned in your initial post (not reproducing it here again for brevity)

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service

I checked the logs and no warnings showed up. Next I checked via the API:

curl http://localhost:9200/_nodes?filter_path=**.mlockall\&pretty

This produced:

{
  "nodes" : {
    "-Xe_hH7ZSC6fp6ZF7UkngQ" : {
      "process" : {
        "mlockall" : true
      }
    }
  }
}

So it looks like some setting is not picked up in your environment. Did you try to reboot the machine?

Daniel


#5

That's really strange. I've tried rebooting the machine and setting the ulimit by hand on the current session too and it works to no avail. If I remove the mlockall it starts up just fine but when I try to enable it then elasticsearch fails.


(Daniel Mitterdorfer) #6

Hi @Mormaii,

one more possibility is that your tmp directory is mounted with noexec so you can try set a different tmp directory for Elasticsearch by adding this to config/jvm.options:

-Djava.io.tmpdir=/path/to/temp/dir

Daniel


#7

I tried setting the the tmpdir to the same data/log dir that elasticsearch uses and I still get the same error

node validation exception bootstrap checks failed
memory locking requested for elasticsearch process but memory is not locked

(Jörg Prante) #8

Do you work over OpenSSH and /etc/ssh/sshd_config contains UsePAM no?


#9

I work over OpenSSH on an EC2 instance. UsePAM is set to yes.


(Jörg Prante) #10

So https://bugzilla.redhat.com/show_bug.cgi?id=1308423 is not of any relevance, I guess.


(Jason Tedor) #11

Have you verified there are no conflicting configurations in any conf files in /etc/security/limits.d/?


#12

There's no files on /etc/security/limits.d/


(system) #13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.