ECK Operator Disable Swap and mlockall:true

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hi,

I want to disable swapiness and bootstrap.memory_lock: true but node is failing. Here is my script:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: prod
spec:
  secureSettings:
  - secretName: gcs-credentials
  http:
    service:
      spec:
        type: LoadBalancer
    tls:
      selfSignedCertificate:
        subjectAltNames:
        - ip: x.x.x.x
  version: 7.9.1
  nodeSets:
  - name: master-nodes
    count: 1
    config:
      node.master: true
      node.data: false
      node.ingest: true
      node.ml: true
      xpack.ml.enabled: true
      cluster.remote.connect: false
      bootstrap.memory_lock: true
    podTemplate:
      spec:
        initContainers:
        - name: install-plugins
          command:
          - sh
          - -c
          - |
            bin/elasticsearch-plugin install --batch repository-gcs
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: ssd
2

Hi @Fikrat_Karimli, thanks for your question.

Can you also provide logs from the Elasticsearch node? You can get them with kubectl logs prod-es-master-nodes-0. These logs should state what was the issue - it might not be related to ECK at all. You can take a look at a similar issue to see if it helps in your case.

3

"Unable to lock JVM Memory: error=12, reason=Cannot allocate memory"

4

Hi,

Swap is usually disabled on K8S, is there any reason to additionally lock the heap in memory through mlockall ?

Maybe that your issue comes from the fact that the Pod is missing the IPC_LOCK capability, could you try to add it to the Pod's security context:

        securityContext:
          capabilities:
            add:
              - IPC_LOCK
5

If it is disabled by default, do I still need to lock memory?

6

If there is no swap then memory pages can't be swapped out to disk, so I think locking memory is useless in this context.

7

Thank you so much for your help!