Search scroll high off-heap memory usage

Elastic Stack Elasticsearch
1

Cluster:

  • Elasticsearch v7.17.1 running with elastic docker image on kubernetes
  • 6 data nodes: 4 CPUs / 14 Go HEAP / 28 Go memory
  • Indice: 3 primary shards (+ 1 replica) , 60 Go data

Usage:

  • nightly full index export, with Apache Spark using elastic for hadoop lib
  • scroll size: 1200
  • 6 tasks in //

Monitoring:

Capture d’écran 2022-03-28 à 10.29.28
Capture d’écran 2022-03-28 à 10.29.282856×472 141 KB

Capture d’écran 2022-03-28 à 10.29.03
Capture d’écran 2022-03-28 à 10.29.032814×510 104 KB

Capture d’écran 2022-03-28 à 10.30.06
Capture d’écran 2022-03-28 à 10.30.061920×456 79 KB

Question:

All data nodes consume +10Go memory during the long search scroll phase, the heap seems fine, how can I debug what is in this memory please?

This is a problem for us because Spark is running on the same k8s cluster and consume also lot of memory.

2

It's likely the OS caching the underlying files that are being requested by Elasticsearch.

3

Thanks you, I will update our Memory Usage graphs !

BTW, is off-heap Elasticsearch usage reported somewhere?

4

Well I am not sure, our infra guys told me page cache is not reported as WSS or RSS, I can graph it and I can see this is not related

Capture d’écran 2022-03-28 à 15.47.18
Capture d’écran 2022-03-28 à 15.47.181920×782 64.3 KB

  • blue = cache
  • yellow = RSS

So the truth is out there :smile:

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.