Kubernetes: cannot allocate memory during bootstrap

Elastic Stack Elasticsearch
1

Hi,

I am trying to get an elasticsearch cluster running in kubernetes.

I am getting following bootstrap error:

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{"type": "server", "timestamp": "2019-06-19T15:38:07,779+0000", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "poc", "node.name": "poc-es-master-2",  "message": "Unable to lock JVM Memory: error=12, reason=Cannot allocate memory"  }
{"type": "server", "timestamp": "2019-06-19T15:38:07,782+0000", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "poc", "node.name": "poc-es-master-2",  "message": "This can result in part of the JVM being swapped out."  }
{"type": "server", "timestamp": "2019-06-19T15:38:07,782+0000", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "poc", "node.name": "poc-es-master-2",  "message": "Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536"  }
{"type": "server", "timestamp": "2019-06-19T15:38:07,786+0000", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "poc", "node.name": "poc-es-master-2",  "message": "These can be adjusted by modifying /etc/security/limits.conf, for example: \n\t# allow user 'elasticsearch' mlockall\n\telasticsearch soft memlock unlimited\n\telasticsearch hard memlock unlimited"  }

I can only bring the nodes up, if I explicitly set bootstrap.memory_lock to false.

In native docker I was able to set --ulimit memlock=-1:-1 and --ulimit nofile=65536:65536 via parameter during docker run.

But how can I do this in kubernetes?

My statefulset configuration looks like this:

# statefulset
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: poc-es-master
  labels:
    role: master
    cluster: poc
spec:
  selector:
    matchLabels:
      app: elasticsearch
      role: master
      cluster: poc
  serviceName: poc-es-master
  replicas: 3
  template:
    metadata:
      labels:
        app: elasticsearch
        role: master
        cluster: poc
    spec:
      initContainers:
      - name: init-sysctl
        image: busybox:1.27.2
        command:
        - sysctl
        - -w
        - vm.max_map_count=262144
        securityContext:
          privileged: true
      containers:
      - name: es-master
        image: docker-registry:443/docker.elastic.co/elasticsearch/elasticsearch:7.1.1
        env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: node.name
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: cluster.name
          value: "poc"
        - name: discovery.seed_hosts
          value: "poc-es-master-0.poc-es-master, poc-es-master-1.poc-es-master, poc-es-master-2.poc-es-master"
        - name: cluster.initial_master_nodes
          value: "poc-es-master-0, poc-es-master-1, poc-es-master-2"
        #- name: discovery.type
        #  value: single-node
        #- name: CLUSTER_NAME
        #  valueFrom:
        #    fieldRef:
        #      fieldPath: labels.cluster
        - name: node.master
          value: "true"
        - name: node.ingest
          value: "true"
        - name: node.data
          value: "true"
        - name: ES_JAVA_OPTS
          value: "-Xms1g -Xmx1g"
        - name: bootstrap.memory_lock
          value: "true"
        - name: network.host
          value: "0.0.0.0"
        - name: PROCESSORS
          valueFrom:
            resourceFieldRef:
              resource: limits.cpu
        resources:
          requests:
            cpu: 0.25
            memory: 1Gi
          limits:
            cpu: 2
            memory: 4Gi
        ports:
        - containerPort: 9300
          name: transport
        - containerPort: 9200
          name: http
        livenessProbe:
          tcpSocket:
            port: transport
          initialDelaySeconds: 20
          periodSeconds: 10
        volumeMounts:
        - name: storage
          mountPath: /usr/share/elasticsearch/data
  volumeClaimTemplates:
  - metadata:
      name: storage
    spec:
      storageClassName: standard-unsafe-no-replica
      accessModes: [ ReadWriteOnce ]
      resources:
        requests:
          storage: 10Gi

Thanks a lot,
Andreas

2

Hi, can anyone please support here?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.