Memory lock check on Elastic Cloud

Elastic Stack Elasticsearch
#1

I'm curious how it is possible that a cluster on Elastic Cloud passes the memory lock bootstrap check even though the memory doesn't seem to be locked.

Here is the mlockall state on all nodes

GET _nodes?filter_path=**.mlockall
{
  "nodes" : {
    "9jimFRh_SpCU6zWrdzb2cA" : {
      "process" : {
        "mlockall" : false
      }
    },
    "7NRLpZlnRjuvww7vtV8AQw" : {
      "process" : {
        "mlockall" : false
      }
    },
    "UHJ96_PzTSmwAVbbLAc5xQ" : {
      "process" : {
        "mlockall" : false
      }
    },
    ...
}

And here is the default cluster setting for bootstrap.memory_lock:

GET _cluster/settings?include_defaults&filter_path=**.memory_lock
{
  "defaults" : {
    "bootstrap" : {
      "memory_lock" : "false"
    }
  }
}

Is it because swappiness and/or swapoff is/are enough?

I'd be happy if anyone has an idea about this. Thanks in advance

#2

No, it's because memory locking is disabled. From the docs you linked:

The memory lock check verifies that if the bootstrap.memory_lock setting is enabled, that the JVM was successfully able to lock the heap.

Since bootstrap.memory_lock is not enabled, this check does nothing.

#3

Thank you David.
What I don't understand is how it is possible that a production cluster on Elastic Cloud can start even though the memory lock bootstrap check is not satisfied?

According to the bootstrap checks description:

If Elasticsearch is in production mode, any bootstrap checks that fail will cause Elasticsearch to refuse to start.

#4

Maybe it helps to look at the source code?

In Cloud, the bootstrap.memory_lock setting is not set, so BootstrapSettings.MEMORY_LOCK_SETTING.get(context.settings()) returns its default, which is false, so it doesn't matter what isMemoryLocked() returns, the check is always satisfied.

#5

You're right, I could have checked the source code first.
Thanks for the insights!

1 Like
#6

@DavidTurner Maybe you can shed some more light on why it is recommended to lock memory and Elastic Cloud doesn't?

1 Like
#7

The recommendation is to disable swap entirely:

There are three approaches to disabling swapping. The preferred option is to completely disable swap. If this is not an option, whether or not to prefer minimizing swappiness versus memory locking is dependent on your environment.

bootstrap.memory_lock is a poor substitute for fixing the OS-level configuration to prevent swapping, because it does not prevent swapping of direct (off-heap) memory.

1 Like
#8

Ok, so that means on Elastic Cloud, this is done at the OS-level using swapoff and it's not necessarily apparent in the cluster/node configuration.

That makes perfect sense then. Thank you so much!!

1 Like