How to debug why memory is not locking in ES5?

[Aivars_Irmejs]

I was forced to change a server recently and I upgraded to ES5.2.0 in process. I'm trying to lock memory and although there are no errors in the log, the memory does not seem to be locked at all.

Process command line looks like this:

/usr/bin/java -Xms1300m -Xmx1300m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/opt/elasticsearch-5.2.0 -cp /opt/elasticsearch-5.2.0/lib/elasticsearch-5.2.0.jar:/opt/elasticsearch-5.2.0/lib/* org.elasticsearch.bootstrap.Elasticsearch -d

I don't have "Unable to lock JVM Memory" in log (I did have it before settings limits.conf).

curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty' returns:
{
"nodes" : {
"g-ioub2QRcuWjL4DRBcuxw" : {
"process" : {
"mlockall" : true
}
}
}
}

What else could I be missing? Why is there no error but the memory still does not lock? Could the error be somewhere else? Thanks.

[jprante (Jörg Prante)]

Can you explain how you determine the process memory is not locked? What operating system is it? What kind of server?

[Aivars_Irmejs]

Thanks for replying.

The server has KiB Mem : 3407872 total. I have -Xms1300m -Xmx1300m configured, yet ES process uses 60% memory at this moment (and it's rising).

OS = CentOS Linux release 7.3.1611, it's a VPS.

[jprante (Jörg Prante)]

So it's still not clear how you determined whether the memory of the ES process is locked or not.

Maybe you have a misunderstanding regarding mlockall. It means large parts of the ES process reside in memory and does not swap out to disk - which is not clear from your info.

Use the command cat /proc/<espid>/status | grep VmLck where <espid> is the ES JVM process ID to determine the amount of process memory locked.

[Aivars_Irmejs]

This is what I get:

cat /proc/1861/status | grep VmLck
VmLck: 1703668 kB

It should be less than that, shouldn't it?