Cannot set up mlockall 'true' on RedHat 6.6

Elastic Stack Elasticsearch
1

I read a lot around, but I'm not able to set mlockall as 'true' yet.
Can you please help me?

This is the procedure I followed so far:

elasticsearch config
# vim /etc/elasticsearch/elasticsearch.yml

bootstrap.mlockall: true

added memlock unlimited in system security limits
# vim /etc/security/limits.conf

elasticsearch - memlock unlimited
root - memlock unlimited```

add these lines for sysconfig
```# vim /etc/sysconfig/elasticsearch```

```ES_HEAP_SIXE=8g
MAX_OPEN_FILES=65535
MAX_LOCKED_MEMORY=unlimited```

uncomment also this line for systemd (even if I'm using SystemV)
```# vim /usr/lib/systemd/system/elasticsearch.service```

```(uncomment)
LimitMEMLOCK=infinity```

to be sure I also added "ulimit -l unlimited" when start elasticsearch the service
```# vim /etc/init.d/elasticsearch```

```(add)
 su $ES_USER --shell /bin/bash -c "ulimit -l unlimited"```

changed the TMP directory to be sure is not mounted as 'noexec'
```# mkdir /opt/apps/elasticsearch/tmp```
```# vim /etc/sysconfig/elasticsearch```

```(change tmp directory)```
```#WORK_DIR=/tmp/elasticsearch
WORK_DIR=/opt/apps/elasticsearch/tmp```
	
start the service
```# service elasticsearch start```

check heap size (OK - uses 8Gb)
```# ps aux | grep elastic
497      26547  200  1.0 11122692 167744 ?     Sl   10:47   0:04 /usr/bin/java -Xms8g -Xmx8g -Djava.awt.headless``


check mlockall (still false)
```# curl http://localhost:9200/_nodes/process?pretty | grep mlockall```

        "mlockall" : false
        "mlockall" : false
        "mlockall" : false
        "mlockall" : false
        "mlockall" : false
        "mlockall" : false
        "mlockall" : false


Thanks
2

running elasticsearch 1.5.2 with java8. what I did is uncomment the bootstrap.mlockall: true and restart the node. other settings pretty much centos stock settings.

[user@localhost ~]$ curl 'localhost:9200/_cat/health?v'
epoch      timestamp cluster       status node.total node.data shards pri relo init unassign pending_tasks 
1432186260 07:31:00  elasticsearch green           3         3      0   0    0    0        0             0 
[user@localhost ~]$ curl -s http://localhost:9200/_nodes/process?pretty | grep mlockall
        "mlockall" : true
        "mlockall" : true
        "mlockall" : true
3

Hi Jason,
when you say 'restart the node' do you mean 'restart elasticsearch' or 'restart the machine'?

4

sorry, my bad, I meant to say restart elasticsearch instance.

5

It's what I did at the beginning:

  • uncomment "mlockall" : true in elasticsearch.yml
  • restart elasticsearch

But It didn't work!
That's why I started changing other settings ...

I'm running elasticsearch 1.5.0 (installed with elasticsearch-1.5.0.noarch.rpm) and Java8.

Any other idea?
Thanks

6

It's a known issue - https://github.com/elastic/elasticsearch/issues/9357

7

After changing all the options above before..

it worked for me after setting up also the WORK_DIR on a different path than /tmp ( because the /tmp directory is mounted as 'noexec') and appending "-Djna.tmpdir=$WORK_DIR" as additional parameter in elasticsearch start command line inside /etc/init.d/elasticsearch ( even if there is already a parameter like "-Des.default.path.work=$WORK_DIR" ).

# curl http://localhost:9200/_nodes/process?pretty | grep mlockall

"mlockall" : true
"mlockall" : false
"mlockall" : true
"mlockall" : false
"mlockall" : true
"mlockall" : true
"mlockall" : true

But I think It's all good because 2 nodes are client nodes (master and data false).

8

Hi guys,

I have same problem.

my system information
Yosemite 10.10. 6
java version "1.8.0_60-ea"

elasticsearch config file

bootstrap.mlockall: true

when I run the 'http://localhost:9200/_nodes/process?pretty' command I get the following output

curl 'http://localhost:9200/_nodes/process?pretty'
{
  "cluster_name" : "elasticsearch",
  "nodes" : {
    "l70PZsNpTTOTFo0nxoZBnA" : {
      "name" : "Eliminator",
      "transport_address" : "inet[192.168.1.21/192.168.1.21:9300]",
      "host" : "terrasacer-air.local",
      "ip" : "192.168.1.21",
      "version" : "1.7.1",
      "build" : "b88f43f",
      "http_address" : "inet[/192.168.1.21:9200]",
      "process" : {
        "refresh_interval_in_millis" : 1000,
        "id" : 8565,
        "max_file_descriptors" : 10240,
        "mlockall" : false
      }
    }
  }
}

but when I run the following command

curl 'http://localhost:9200/_nodes/?pretty'
{
...
,
  "foreground" : "yes",
   "bootstrap" : {
          "mlockall" : "true"
  },
...
}

what is going on? :smile:

9

Mac OS X does not have mlockall implemented.

10

Hi @jprante,

First, thank you for your answer.

The documentation says:

The third option is to use mlockall on Linux/Unix systems, or VirtualLock on Windows, to try to lock the process address space into RAM, preventing any Elasticsearch memory from being swapped out. This can be done, by adding this line to the config/elasticsearch.yml file:

Is there misdirection here?

11

The documentation is correct.

Apple decided not to expose the memory pinning functions to user space processes, like mlockall. Only Mac OS X kernel drivers are allowed to do this.

1 Like
12

Okey, I understand.

In this case, misleading information returned by the curl 'http://localhost:9200/_nodes/?pretty' command:

curl 'http://localhost:9200/_nodes/?pretty'
{
...
,
  "foreground" : "yes",
   "bootstrap" : {
          "mlockall" : "true"
  },
...
}
13

These are just your command line arguments that you have passed to the JVM. The output means nothing with regard to having the parameters applied successfully.

1 Like
14

Ok @jprante,

Thank you very much for the explanation.