Limit Elasticsearch to 512 MB on Kubernetes

[lucbas (Lucky)]

Hi all

I'm trying to set up elastic search on Kubernetes (GKE) and try to limit its memory usage to max 512MB (or even lower).
Performance is not that crucial for us, we just need to run some Jaeger queries from time to time for tracing purposes.
When Jaeger tried to access ES, it crashes.
I tried to set the Heap to 256MB and the limit to 512MB in the manifest.

Currently, this is the config YAML:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
spec:
  version: 7.7.0
  nodeSets:
  - name: default
    count: 1
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          env:
          - name: ES_JAVA_OPTS
            value: -Xms256m -Xmx256m
          resources:
            requests:
              memory: 512Mi
              cpu: 0.2
            limits:
              memory: 512Mi
              cpu: 0.2
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false

Would appreciate any help!

[Christian_Dahlqvist]

I know there ha been work on making Elasticsearch work better in low memory environments, but I do not think I have seek Elasticsearch do anything useful with less that 512MB of heap. That is the smallest I would recommend setting up.

[lucbas (Lucky)]

Hi Christian

Many thanks for your answer!

I would like to avoid reserving 1 gig for the ES container with a heap of 512 if it's possible.
Maybe there are some tricks that could help?

Either way, the following config should be workable I assume:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
spec:
  version: 7.7.0
  nodeSets:
  - name: default
    count: 1
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          env:
          - name: ES_JAVA_OPTS
            value: -Xms512m -Xmx512m
          resources:
            requests:
              memory: 1Gi
              cpu: 0.2
            limits:
              memory: 1Gi
              cpu: 0.2
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false

[Mikhail_Khludnev]

When Jaeger tried to access ES, it crashes.

Hi,
How it crashes specifically? Are there any messages? Does it happen inside of container, or there some logs in docker host?

[lucbas (Lucky)]

Hi Mikhail

The container does not seem to get ready due to the readiness probe. This is what the ES pod shows:

{"timestamp": "2020-05-29T16:40:20+0000", "message": "readiness probe failed", "curl_rc": "28"}
{"timestamp": "2020-05-29T16:40:51+0000", "message": "readiness probe failed", "curl_rc": "28"}

[lucbas (Lucky)]

When talking about resource optimizations, what is the recommended way to reduce the cpu/memory requests and limits of the ES operator?