High RAM usage by data nodes (Kubernetes)

itsron · January 23, 2021, 8:45pm

We are running v7.10.1 on Kubernetes. We've 4 data nodes each with 4 cpus and 16gb RAM (8gb heap). Currently it has only metricbeat and filebeat writing data into it which is a total of ~1.5M docs/hour (~10GB/day,retained for 1 week). These nodes do not run anything else other than a couple of essential system pods (which do not consume any significant resources).

I've noticed that the RAM usage is >70% even when there are no queries. Why is that?

id   name                   r         m      hc hp     rc rp   rm cpu     dt      du     da load_1m load_5m
gb9i elasticsearch-data-0   cdhilrstw -   4.3gb 54 12.4gb 78 16gb   5 78.6gb   3.5gb   75gb    0.16    0.29
A8zB elasticsearch-data-1   cdhilrstw -   2.4gb 30 12.2gb 77 16gb   5 78.6gb   3.3gb 75.2gb    0.72    0.28
a-Lq elasticsearch-data-2   cdhilrstw -   2.3gb 29  9.9gb 62 16gb   3 78.6gb   1.1gb 77.4gb    0.16    0.14
Xa15 elasticsearch-data-3   cdhilrstw -   3.7gb 47  9.7gb 61 16gb   3 78.6gb 962.8mb 77.6gb    0.05    0.05
T3s2 elasticsearch-master-0 lm        * 706.3mb 17  4.4gb 56  8gb   6  4.7gb  36.4mb  4.7gb    0.00    0.05
0ile elasticsearch-master-1 lm        -   1.9gb 47  4.4gb 56  8gb   2  4.7gb  36.4mb  4.7gb    0.07    0.13
Cbi_ elasticsearch-master-2 lm        - 186.7mb  4  4.4gb 56  8gb   3  4.7gb  36.4mb  4.7gb    0.10    0.08
e5E9 elasticsearch-master-3 lm        -   1.4gb 35  4.4gb 55  8gb   3  4.7gb  36.5mb  4.7gb    0.48    0.37

_cat/shards output:

index                               shard prirep state      docs   store node
.slm-history-3-000001               0     r      STARTED                 elasticsearch-data-0
.slm-history-3-000001               0     p      STARTED                 elasticsearch-data-2
.kibana_security_session_1          0     r      STARTED                 elasticsearch-data-1
.kibana_security_session_1          0     p      STARTED                 elasticsearch-data-2
ilm-history-3-000001                0     r      STARTED                 elasticsearch-data-3
ilm-history-3-000001                0     p      STARTED                 elasticsearch-data-1
.apm-agent-configuration            0     p      STARTED       0    208b elasticsearch-data-0
.apm-agent-configuration            0     r      STARTED       0    208b elasticsearch-data-2
.apm-custom-link                    0     p      STARTED       0    208b elasticsearch-data-3
.apm-custom-link                    0     r      STARTED       0    208b elasticsearch-data-1
.tasks                              0     p      STARTED       1   6.9kb elasticsearch-data-1
.tasks                              0     r      STARTED       1   6.9kb elasticsearch-data-2
.kibana-event-log-7.10.1            0     p      STARTED       1   6.9kb elasticsearch-data-0
.kibana-event-log-7.10.1            0     r      STARTED       1   6.9kb elasticsearch-data-3
.kibana_task_manager_1              0     r      STARTED       6 367.7kb elasticsearch-data-0
.kibana_task_manager_1              0     p      STARTED       6 362.2kb elasticsearch-data-1
.kibana_1                           0     p      STARTED       6   8.3kb elasticsearch-data-3
.kibana_1                           0     r      STARTED       6   8.3kb elasticsearch-data-1
.async-search                       0     p      STARTED       9   5.8mb elasticsearch-data-0
.async-search                       0     r      STARTED       9   5.8mb elasticsearch-data-2
.kibana_2                           0     r      STARTED      32   4.4mb elasticsearch-data-3
.kibana_2                           0     p      STARTED      32   2.8mb elasticsearch-data-2
.security-7                         0     r      STARTED      56   193kb elasticsearch-data-0
.security-7                         0     p      STARTED      56   193kb elasticsearch-data-1
logxx-xxxx-xxx-data-2021.01.23      0     p      STARTED   16993     7mb elasticsearch-data-0
logxx-xxxx-xxx-data-2021.01.23      0     r      STARTED   16993     7mb elasticsearch-data-2
metricbeat-8.0.0-2021.01.23-000001  0     p      STARTED   26791  14.8mb elasticsearch-data-0
metricbeat-8.0.0-2021.01.23-000001  0     r      STARTED   26791  14.9mb elasticsearch-data-3
logxx-xxxx-xxx-data-2021.01.16      0     r      STARTED   84672  31.6mb elasticsearch-data-1
logxx-xxxx-xxx-data-2021.01.16      0     p      STARTED   84672  31.6mb elasticsearch-data-2
logxx-xxxx-xxx-data-2021.01.21      0     p      STARTED   84960  24.8mb elasticsearch-data-3
logxx-xxxx-xxx-data-2021.01.21      0     r      STARTED   84960  24.8mb elasticsearch-data-2
logxx-xxxx-xxx-data-2021.01.15      0     r      STARTED  223695  58.4mb elasticsearch-data-0
logxx-xxxx-xxx-data-2021.01.15      0     p      STARTED  223695  58.4mb elasticsearch-data-1
logxx-xxxx-xxx-data-2021.01.14      0     r      STARTED  230341  57.2mb elasticsearch-data-3
logxx-xxxx-xxx-data-2021.01.14      0     p      STARTED  230341  57.2mb elasticsearch-data-1
logxx-xxxx-xxx-data-2021.01.22      0     p      STARTED  509953 172.3mb elasticsearch-data-0
logxx-xxxx-xxx-data-2021.01.22      0     r      STARTED  509953 172.3mb elasticsearch-data-3
logxx-xxxx-xxx-data-2021.01.19      0     p      STARTED  651965 152.4mb elasticsearch-data-0
logxx-xxxx-xxx-data-2021.01.19      0     r      STARTED  651965 152.4mb elasticsearch-data-2
logxx-xxxx-xxx-data-2021.01.18      0     r      STARTED  670791 138.2mb elasticsearch-data-3
logxx-xxxx-xxx-data-2021.01.18      0     p      STARTED  670791 138.2mb elasticsearch-data-2
logxx-xxxx-xxx-data-2021.01.20      0     r      STARTED  765555 159.7mb elasticsearch-data-3
logxx-xxxx-xxx-data-2021.01.20      0     p      STARTED  765555 159.7mb elasticsearch-data-1
filebeat-7.10.1-2021.01.22-000001   0     r      STARTED  881964 292.5mb elasticsearch-data-3
filebeat-7.10.1-2021.01.22-000001   0     p      STARTED  882031 317.3mb elasticsearch-data-2
metricbeat-7.10.1-2021.01.22-000001 0     p      STARTED 7443102   2.7gb elasticsearch-data-1
metricbeat-7.10.1-2021.01.22-000001 0     r      STARTED 7443119   2.6gb elasticsearch-data-0

_cat/indices output:

health status index                               uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   metricbeat-7.10.1-2021.01.22-000001 c0QgKkR0RKyLyALbO4gHJA   1   1    7476803            0      5.4gb          2.7gb
green  open   metricbeat-8.0.0-2021.01.23-000001  _ey3bMu1Qt6uGrw1MkR_aQ   1   1      26791            0     29.7mb         14.8mb
green  open   .apm-agent-configuration            xm1KHwtOTrWZa4pIGtiwaQ   1   1          0            0       416b           208b
green  open   logxx-xxxx-xxx-data-2021.01.21      -4VTtMV-RG26QmDgBr4Gyg   1   1      84960            0     49.6mb         24.8mb
green  open   logxx-xxxx-xxx-data-2021.01.20      v4us1dBZRe69o7yxPONmRA   1   1     765555            0    319.5mb        159.7mb
green  open   logxx-xxxx-xxx-data-2021.01.23      4-KqF7EhSfGe0aGNIyhckw   1   1      16993            0     14.1mb            7mb
green  open   .kibana_2                           5fz9INfARTOnPe3ukAb-Ig   1   1         33            0      8.8mb          4.4mb
green  open   logxx-xxxx-xxx-data-2021.01.22      irSvCD9cQ2uvBHkvpMSSEg   1   1     509953            0    344.7mb        172.3mb
green  open   .kibana_1                           EIUEMtDzSrePftYkAfTbEA   1   1          6            0     16.7kb          8.3kb
green  open   logxx-xxxx-xxx-data-2021.01.14      wNB3aaBUSnKSQl_Be8kEsw   1   1     230341            0    114.4mb         57.2mb
green  open   .tasks                              nXkgUyYrRn-NBJqFYoj5iA   1   1          1            0     13.9kb          6.9kb
green  open   logxx-xxxx-xxx-data-2021.01.16      RSKmL8nLSCCgNa_PSM8j9w   1   1      84672            0     63.2mb         31.6mb
green  open   logxx-xxxx-xxx-data-2021.01.15      sSEJDbZqR8yPe8n0grFbxw   1   1     223695            0    116.9mb         58.4mb
green  open   .security-7                         JpH0FVXsSJi5RNg-vnBszg   1   1         56            7    386.1kb          193kb
green  open   logxx-xxxx-xxx-data-2021.01.18      DBern0wiRkmRnaGKpUuZ_g   1   1     670791            0    276.5mb        138.2mb
green  open   .kibana-event-log-7.10.1            QiZQazsfS_e-3xx8tYW-nw   1   1          1            0     13.9kb          6.9kb
green  open   .apm-custom-link                    N7ZXpvn4R6S5591BkS79qA   1   1          0            0       416b           208b
green  open   .kibana_task_manager_1              FcDffMGUTJCesl34DGx_uw   1   1          6         3729    769.2kb        381.9kb
green  open   logxx-xxxx-xxx-data-2021.01.19      19U6OoR0ShqMXb0xQyRgAQ   1   1     651965            0    304.9mb        152.4mb
green  open   .async-search                       lantDqGURdq5AXXf4lRikA   1   1          9            9     11.6mb          5.8mb
green  open   filebeat-7.10.1-2021.01.22-000001   1VRkhAA1R8ONXECuknyAxw   1   1     885123            0    626.4mb        318.5mb

warkolm · January 24, 2021, 11:46pm

Welcome to our community!

It is possibly/probably the OS caching files that are being heavily used/written to in memory.

itsron · January 25, 2021, 3:44pm

Thanks for your reply. I used free -m -g to see the memory usage:

                    total        used        free      shared  buff/cache   available
    Mem:             15           9           0           0           5           5
    Swap:             0           0           0

which shows me that there is 5Gb of buff/cache and used+buff/cache roughly equal to what I see in /cat_nodes output.

Do you think 16GB of RAM is not enough for the amount of data that I'm generating? Currently it is about 21M/day (~10-12GB of data) with 1 Primary/ 1 replica on a 4 data node cluster. I want to retain this data max 1 week.

Sorry if I sound stupid, I'm new to elasticsearch.

itsron · January 26, 2021, 3:49pm

@warkolm Also I can see even masters take 4.5Gb of RAM and has lots of page caches. Is this normal? I imagined masters shouldn't use so much resources.

warkolm · January 26, 2021, 7:58pm

There's (basically) two memory "areas" when talking about use in Elasticsearch.

There's the primary one which is heap, which is what you tell the JVM it's allowed to use and is what you can directly influence the size of.
Then there's off-heap memory, which is mostly the OS managing commonly paged files to improve performance. You can manage this, but it's outside the scope of what we can help with.

You really want to focus on your heap use, if you aren't getting heavy garbage collection, circuit breaker responses or OOM (which will be rare), or seeing slowness at all (which happens with a full heap) then you should be fine.

system · February 23, 2021, 7:58pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.