Adjusting limits on elasticsearch containers

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hello,
We applied a YML file that looks like this:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
  namespace: elastic-stack
spec:
  version: 7.6.1
  nodeSets:
    - name: default
      count: 3
      config:
        node.master: true
        node.data: true
        node.ingest: true
        node.store.allow_mmap: false
      podTemplate:
        spec:
          initContainers:
          - name: install-plugins
            command:
            - sh
            - -c
            - |
              bin/elasticsearch-plugin install --batch repository-gcs
      metadata:
        labels:
        spec:
          # this changes the kernel setting on the node to allow ES to use mmap
          # if you uncomment this init container you will likely also want to remove the
          # "node.store.allow_mmap: false" setting above
          # initContainers:
          # - name: sysctl
          #   securityContext:
          #     privileged: true
          #   command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
          containers:
            - name: elasticsearch
              # specify resource limits and requests
              resources:
                limits:
                  memory: 8Gi
                  cpu: 4
              env:
                - name: ES_JAVA_OPTS
                  value: "-Xms2g -Xmx2g"
      # request 2Gi of persistent data storage for pods in this topology element
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 200Gi
            storageClassName: standard
  http:
    service:
      spec:
        # expose this cluster Service with a LoadBalancer
        ports:
          - name: http
            port: 9200
            targetPort: 9200
        type: ClusterIP
    tls:
      certificate:
        secretName: elasticsearch-tls-cert

My understanding from reading the documentation here was that the resource limits here would set the resource limits on the individual Elasticsearch containers running in our cluster. But when I kubectl describe the pods, I see that they have a different memory limit set:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    update.k8s.elastic.co/timestamp: "2022-03-18T20:22:43.84655766Z"
  creationTimestamp: "2022-03-18T20:13:54Z"
  generateName: elasticsearch-es-default-
  labels:
    common.k8s.elastic.co/type: elasticsearch
    controller-revision-hash: elasticsearch-es-default-b8487bb44
    elasticsearch.k8s.elastic.co/cluster-name: elasticsearch
    elasticsearch.k8s.elastic.co/config-hash: "3850528256"
    elasticsearch.k8s.elastic.co/http-scheme: https
    elasticsearch.k8s.elastic.co/node-data: "true"
    elasticsearch.k8s.elastic.co/node-ingest: "true"
    elasticsearch.k8s.elastic.co/node-master: "true"
    elasticsearch.k8s.elastic.co/node-ml: "true"
    elasticsearch.k8s.elastic.co/statefulset-name: elasticsearch-es-default
    elasticsearch.k8s.elastic.co/version: 7.6.1
    statefulset.kubernetes.io/pod-name: elasticsearch-es-default-0
  name: elasticsearch-es-default-0
  namespace: elastic-stack
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: elasticsearch-es-default
    uid: cd3daa4a-7004-4ae3-a288-46b0ddfe19fe
  resourceVersion: "12101078"
  uid: 9551f75e-7ec6-497f-ad70-413da314a80b
spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchLabels:
              elasticsearch.k8s.elastic.co/cluster-name: elasticsearch
          topologyKey: kubernetes.io/hostname
        weight: 100
  automountServiceAccountToken: false
  containers:
  - env:
    - name: HEADLESS_SERVICE_NAME
      value: elasticsearch-es-default
    - name: NSS_SDB_USE_CACHE
      value: "no"
    - name: POD_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: PROBE_PASSWORD_PATH
      value: /mnt/elastic-internal/probe-user/elastic-internal-probe
    - name: PROBE_USERNAME
      value: elastic-internal-probe
    - name: READINESS_PROBE_PROTOCOL
      value: https
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.1
    imagePullPolicy: IfNotPresent
    lifecycle:
      preStop:
        exec:
          command:
          - bash
          - -c
          - /mnt/elastic-internal/scripts/pre-stop-hook-script.sh
    name: elasticsearch
    ports:
    - containerPort: 9200
      name: http
      protocol: TCP
    - containerPort: 9300
      name: transport
      protocol: TCP
    readinessProbe:
      exec:
        command:
        - bash
        - -c
        - /mnt/elastic-internal/scripts/readiness-probe-script.sh
      failureThreshold: 3
      initialDelaySeconds: 10
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 5
    resources:
      limits:
        memory: 2Gi
      requests:
        memory: 2Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /mnt/elastic-internal/downward-api
      name: downward-api
      readOnly: true
    - mountPath: /usr/share/elasticsearch/bin
      name: elastic-internal-elasticsearch-bin-local
    - mountPath: /mnt/elastic-internal/elasticsearch-config
      name: elastic-internal-elasticsearch-config
      readOnly: true
    - mountPath: /usr/share/elasticsearch/config
      name: elastic-internal-elasticsearch-config-local
    - mountPath: /usr/share/elasticsearch/plugins
      name: elastic-internal-elasticsearch-plugins-local
    - mountPath: /usr/share/elasticsearch/config/http-certs
      name: elastic-internal-http-certificates
      readOnly: true
    - mountPath: /mnt/elastic-internal/probe-user
      name: elastic-internal-probe-user
      readOnly: true
    - mountPath: /mnt/elastic-internal/scripts
      name: elastic-internal-scripts
      readOnly: true
    - mountPath: /usr/share/elasticsearch/config/transport-certs
      name: elastic-internal-transport-certificates
      readOnly: true
    - mountPath: /mnt/elastic-internal/unicast-hosts
      name: elastic-internal-unicast-hosts
      readOnly: true
    - mountPath: /mnt/elastic-internal/xpack-file-realm
      name: elastic-internal-xpack-file-realm
      readOnly: true
    - mountPath: /usr/share/elasticsearch/data
      name: elasticsearch-data
    - mountPath: /usr/share/elasticsearch/logs
      name: elasticsearch-logs
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: elasticsearch-es-default-0
  initContainers:
  - command:
    - bash
    - -c
    - /mnt/elastic-internal/scripts/prepare-fs.sh
    env:
    - name: POD_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: POD_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.1
    imagePullPolicy: IfNotPresent
    name: elastic-internal-init-filesystem
    resources:
      limits:
        cpu: 100m
        memory: 50Mi
      requests:
        cpu: 100m
        memory: 50Mi
    securityContext:
      privileged: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /mnt/elastic-internal/elasticsearch-config-local
      name: elastic-internal-elasticsearch-config-local
    - mountPath: /mnt/elastic-internal/elasticsearch-plugins-local
      name: elastic-internal-elasticsearch-plugins-local
    - mountPath: /mnt/elastic-internal/elasticsearch-bin-local
      name: elastic-internal-elasticsearch-bin-local
    - mountPath: /mnt/elastic-internal/transport-certificates
      name: elastic-internal-transport-certificates
      readOnly: true
    - mountPath: /mnt/elastic-internal/scripts
      name: elastic-internal-scripts
      readOnly: true
    - mountPath: /usr/share/elasticsearch/data
      name: elasticsearch-data
    - mountPath: /usr/share/elasticsearch/logs
      name: elasticsearch-logs
    - mountPath: /mnt/elastic-internal/downward-api
      name: downward-api
      readOnly: true
    - mountPath: /mnt/elastic-internal/elasticsearch-config
      name: elastic-internal-elasticsearch-config
      readOnly: true
    - mountPath: /usr/share/elasticsearch/config/http-certs
      name: elastic-internal-http-certificates
      readOnly: true
    - mountPath: /mnt/elastic-internal/probe-user
      name: elastic-internal-probe-user
      readOnly: true
    - mountPath: /mnt/elastic-internal/unicast-hosts
      name: elastic-internal-unicast-hosts
      readOnly: true
    - mountPath: /mnt/elastic-internal/xpack-file-realm
      name: elastic-internal-xpack-file-realm
      readOnly: true
  - command:
    - sh
    - -c
    - |
      bin/elasticsearch-plugin install --batch repository-gcs
    env:
    - name: POD_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.1
    imagePullPolicy: IfNotPresent
    name: install-plugins
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /mnt/elastic-internal/downward-api
      name: downward-api
      readOnly: true
    - mountPath: /usr/share/elasticsearch/bin
      name: elastic-internal-elasticsearch-bin-local
    - mountPath: /mnt/elastic-internal/elasticsearch-config
      name: elastic-internal-elasticsearch-config
      readOnly: true
    - mountPath: /usr/share/elasticsearch/config
      name: elastic-internal-elasticsearch-config-local
    - mountPath: /usr/share/elasticsearch/plugins
      name: elastic-internal-elasticsearch-plugins-local
    - mountPath: /usr/share/elasticsearch/config/http-certs
      name: elastic-internal-http-certificates
      readOnly: true
    - mountPath: /mnt/elastic-internal/probe-user
      name: elastic-internal-probe-user
      readOnly: true
    - mountPath: /mnt/elastic-internal/scripts
      name: elastic-internal-scripts
      readOnly: true
    - mountPath: /usr/share/elasticsearch/config/transport-certs
      name: elastic-internal-transport-certificates
      readOnly: true
    - mountPath: /mnt/elastic-internal/unicast-hosts
      name: elastic-internal-unicast-hosts
      readOnly: true
    - mountPath: /mnt/elastic-internal/xpack-file-realm
      name: elastic-internal-xpack-file-realm
      readOnly: true
    - mountPath: /usr/share/elasticsearch/data
      name: elasticsearch-data
    - mountPath: /usr/share/elasticsearch/logs
      name: elasticsearch-logs
  nodeName: gke-twobird-elk-stack-default-pool-f6d3efa3-fldm
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  subdomain: elasticsearch-es-default
  terminationGracePeriodSeconds: 180
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: elasticsearch-data
    persistentVolumeClaim:
      claimName: elasticsearch-data-elasticsearch-es-default-0
  - downwardAPI:
      defaultMode: 420
      items:
      - fieldRef:
          apiVersion: v1
          fieldPath: metadata.labels
        path: labels
    name: downward-api
  - emptyDir: {}
    name: elastic-internal-elasticsearch-bin-local
  - name: elastic-internal-elasticsearch-config
    secret:
      defaultMode: 420
      optional: false
      secretName: elasticsearch-es-default-es-config
  - emptyDir: {}
    name: elastic-internal-elasticsearch-config-local
  - emptyDir: {}
    name: elastic-internal-elasticsearch-plugins-local
  - name: elastic-internal-http-certificates
    secret:
      defaultMode: 420
      optional: false
      secretName: elasticsearch-es-http-certs-internal
  - name: elastic-internal-probe-user
    secret:
      defaultMode: 420
      items:
      - key: elastic-internal-probe
        path: elastic-internal-probe
      optional: false
      secretName: elasticsearch-es-internal-users
  - configMap:
      defaultMode: 493
      name: elasticsearch-es-scripts
      optional: false
    name: elastic-internal-scripts
  - name: elastic-internal-transport-certificates
    secret:
      defaultMode: 420
      optional: false
      secretName: elasticsearch-es-transport-certificates
  - configMap:
      defaultMode: 420
      name: elasticsearch-es-unicast-hosts
      optional: false
    name: elastic-internal-unicast-hosts
  - name: elastic-internal-xpack-file-realm
    secret:
      defaultMode: 420
      optional: false
      secretName: elasticsearch-es-xpack-file-realm
  - emptyDir: {}
    name: elasticsearch-logs
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2022-03-18T20:16:30Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2022-03-18T20:16:49Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2022-03-18T20:16:49Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2022-03-18T20:13:54Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://af076627b6aa096fd14973092e7ffd24867f167c065d3089fc86e24556445100
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.1
    imageID: docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:d74a4d9aa241cc7fda110a38f1208869407f1242e93c733281452cd9831a0064
    lastState: {}
    name: elasticsearch
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2022-03-18T20:16:30Z"
  hostIP: 10.138.0.11
  initContainerStatuses:
  - containerID: docker://61624f2862a05f77eaf6408645d12085f901a364bfaef3b2e181cbcb768737db
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.1
    imageID: docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:d74a4d9aa241cc7fda110a38f1208869407f1242e93c733281452cd9831a0064
    lastState: {}
    name: elastic-internal-init-filesystem
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: docker://61624f2862a05f77eaf6408645d12085f901a364bfaef3b2e181cbcb768737db
        exitCode: 0
        finishedAt: "2022-03-18T20:16:23Z"
        reason: Completed
        startedAt: "2022-03-18T20:16:21Z"
  - containerID: docker://e2fa26cc3d51eb27f3a6cc7a49ff23432b739dcda0e26e40e0a66fae7aec7740
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.1
    imageID: docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:d74a4d9aa241cc7fda110a38f1208869407f1242e93c733281452cd9831a0064
    lastState: {}
    name: install-plugins
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: docker://e2fa26cc3d51eb27f3a6cc7a49ff23432b739dcda0e26e40e0a66fae7aec7740
        exitCode: 0
        finishedAt: "2022-03-18T20:16:29Z"
        reason: Completed
        startedAt: "2022-03-18T20:16:24Z"
  phase: Running
  podIP: 10.28.2.5
  podIPs:
  - ip: 10.28.2.5
  qosClass: Burstable
  startTime: "2022-03-18T20:13:54Z"

Sorry, that's a huge YML dump, but the relevant part are the resource limits and requests for memory, which are both set to 2Gi. There is no CPU limit set.

How do I adjust the memory limits for the individual Elasticsearch pods? Is setting this in the Elasticsearch YML I included not the correct way to do this?

2

I did not try to replicate the issue but something looks wrong with the indentation, I guess metadata should be in the podTemplate section ? Also you should have only one spec in the podTemplate. You can refer to the example here.

Hope it helps.

1 Like
3

Thanks, that was the problem! I'm surprised we were able to apply the YAML successfully at all with that mistake.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.