# High RAM usage by data nodes (Kubernetes) **URL:** \ **Category:** Elasticsearch\ **Created:** [January 23, 2021, 8:45pm UTC](https://discuss.elastic.co/t/high-ram-usage-by-data-nodes-kubernetes/262008 "2021-01-23T20:45:58Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![itsron](https://avatars.discourse-cdn.com/v4/letter/i/a88e4f/32.png) [@itsron](https://discuss.elastic.co/u/itsron)\ **Post date:** [January 23, 2021, 8:45pm UTC](https://discuss.elastic.co/t/high-ram-usage-by-data-nodes-kubernetes/262008/1 "2021-01-23T20:45:58Z")
We are running v7.10.1 on Kubernetes. We've 4 data nodes each with 4 cpus and 16gb RAM (8gb heap). Currently it has only metricbeat and filebeat writing data into it which is a total of ~1.5M docs/hour (~10GB/day,retained for 1 week). These nodes do not run anything else other than a couple of essential system pods (which do not consume any significant resources). I've noticed that the RAM usage is \>70% even when there are no queries. Why is that? ``` id name r m hc hp rc rp rm cpu dt du da load_1m load_5m gb9i elasticsearch-data-0 cdhilrstw - 4.3gb 54 12.4gb 78 16gb 5 78.6gb 3.5gb 75gb 0.16 0.29 A8zB elasticsearch-data-1 cdhilrstw - 2.4gb 30 12.2gb 77 16gb 5 78.6gb 3.3gb 75.2gb 0.72 0.28 a-Lq elasticsearch-data-2 cdhilrstw - 2.3gb 29 9.9gb 62 16gb 3 78.6gb 1.1gb 77.4gb 0.16 0.14 Xa15 elasticsearch-data-3 cdhilrstw - 3.7gb 47 9.7gb 61 16gb 3 78.6gb 962.8mb 77.6gb 0.05 0.05 T3s2 elasticsearch-master-0 lm * 706.3mb 17 4.4gb 56 8gb 6 4.7gb 36.4mb 4.7gb 0.00 0.05 0ile elasticsearch-master-1 lm - 1.9gb 47 4.4gb 56 8gb 2 4.7gb 36.4mb 4.7gb 0.07 0.13 Cbi_ elasticsearch-master-2 lm - 186.7mb 4 4.4gb 56 8gb 3 4.7gb 36.4mb 4.7gb 0.10 0.08 e5E9 elasticsearch-master-3 lm - 1.4gb 35 4.4gb 55 8gb 3 4.7gb 36.5mb 4.7gb 0.48 0.37 ``` `_cat/shards` output: ``` index shard prirep state docs store node .slm-history-3-000001 0 r STARTED elasticsearch-data-0 .slm-history-3-000001 0 p STARTED elasticsearch-data-2 .kibana_security_session_1 0 r STARTED elasticsearch-data-1 .kibana_security_session_1 0 p STARTED elasticsearch-data-2 ilm-history-3-000001 0 r STARTED elasticsearch-data-3 ilm-history-3-000001 0 p STARTED elasticsearch-data-1 .apm-agent-configuration 0 p STARTED 0 208b elasticsearch-data-0 .apm-agent-configuration 0 r STARTED 0 208b elasticsearch-data-2 .apm-custom-link 0 p STARTED 0 208b elasticsearch-data-3 .apm-custom-link 0 r STARTED 0 208b elasticsearch-data-1 .tasks 0 p STARTED 1 6.9kb elasticsearch-data-1 .tasks 0 r STARTED 1 6.9kb elasticsearch-data-2 .kibana-event-log-7.10.1 0 p STARTED 1 6.9kb elasticsearch-data-0 .kibana-event-log-7.10.1 0 r STARTED 1 6.9kb elasticsearch-data-3 .kibana_task_manager_1 0 r STARTED 6 367.7kb elasticsearch-data-0 .kibana_task_manager_1 0 p STARTED 6 362.2kb elasticsearch-data-1 .kibana_1 0 p STARTED 6 8.3kb elasticsearch-data-3 .kibana_1 0 r STARTED 6 8.3kb elasticsearch-data-1 .async-search 0 p STARTED 9 5.8mb elasticsearch-data-0 .async-search 0 r STARTED 9 5.8mb elasticsearch-data-2 .kibana_2 0 r STARTED 32 4.4mb elasticsearch-data-3 .kibana_2 0 p STARTED 32 2.8mb elasticsearch-data-2 .security-7 0 r STARTED 56 193kb elasticsearch-data-0 .security-7 0 p STARTED 56 193kb elasticsearch-data-1 logxx-xxxx-xxx-data-2021.01.23 0 p STARTED 16993 7mb elasticsearch-data-0 logxx-xxxx-xxx-data-2021.01.23 0 r STARTED 16993 7mb elasticsearch-data-2 metricbeat-8.0.0-2021.01.23-000001 0 p STARTED 26791 14.8mb elasticsearch-data-0 metricbeat-8.0.0-2021.01.23-000001 0 r STARTED 26791 14.9mb elasticsearch-data-3 logxx-xxxx-xxx-data-2021.01.16 0 r STARTED 84672 31.6mb elasticsearch-data-1 logxx-xxxx-xxx-data-2021.01.16 0 p STARTED 84672 31.6mb elasticsearch-data-2 logxx-xxxx-xxx-data-2021.01.21 0 p STARTED 84960 24.8mb elasticsearch-data-3 logxx-xxxx-xxx-data-2021.01.21 0 r STARTED 84960 24.8mb elasticsearch-data-2 logxx-xxxx-xxx-data-2021.01.15 0 r STARTED 223695 58.4mb elasticsearch-data-0 logxx-xxxx-xxx-data-2021.01.15 0 p STARTED 223695 58.4mb elasticsearch-data-1 logxx-xxxx-xxx-data-2021.01.14 0 r STARTED 230341 57.2mb elasticsearch-data-3 logxx-xxxx-xxx-data-2021.01.14 0 p STARTED 230341 57.2mb elasticsearch-data-1 logxx-xxxx-xxx-data-2021.01.22 0 p STARTED 509953 172.3mb elasticsearch-data-0 logxx-xxxx-xxx-data-2021.01.22 0 r STARTED 509953 172.3mb elasticsearch-data-3 logxx-xxxx-xxx-data-2021.01.19 0 p STARTED 651965 152.4mb elasticsearch-data-0 logxx-xxxx-xxx-data-2021.01.19 0 r STARTED 651965 152.4mb elasticsearch-data-2 logxx-xxxx-xxx-data-2021.01.18 0 r STARTED 670791 138.2mb elasticsearch-data-3 logxx-xxxx-xxx-data-2021.01.18 0 p STARTED 670791 138.2mb elasticsearch-data-2 logxx-xxxx-xxx-data-2021.01.20 0 r STARTED 765555 159.7mb elasticsearch-data-3 logxx-xxxx-xxx-data-2021.01.20 0 p STARTED 765555 159.7mb elasticsearch-data-1 filebeat-7.10.1-2021.01.22-000001 0 r STARTED 881964 292.5mb elasticsearch-data-3 filebeat-7.10.1-2021.01.22-000001 0 p STARTED 882031 317.3mb elasticsearch-data-2 metricbeat-7.10.1-2021.01.22-000001 0 p STARTED 7443102 2.7gb elasticsearch-data-1 metricbeat-7.10.1-2021.01.22-000001 0 r STARTED 7443119 2.6gb elasticsearch-data-0 ``` `_cat/indices` output: ``` health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open metricbeat-7.10.1-2021.01.22-000001 c0QgKkR0RKyLyALbO4gHJA 1 1 7476803 0 5.4gb 2.7gb green open metricbeat-8.0.0-2021.01.23-000001 _ey3bMu1Qt6uGrw1MkR_aQ 1 1 26791 0 29.7mb 14.8mb green open .apm-agent-configuration xm1KHwtOTrWZa4pIGtiwaQ 1 1 0 0 416b 208b green open logxx-xxxx-xxx-data-2021.01.21 -4VTtMV-RG26QmDgBr4Gyg 1 1 84960 0 49.6mb 24.8mb green open logxx-xxxx-xxx-data-2021.01.20 v4us1dBZRe69o7yxPONmRA 1 1 765555 0 319.5mb 159.7mb green open logxx-xxxx-xxx-data-2021.01.23 4-KqF7EhSfGe0aGNIyhckw 1 1 16993 0 14.1mb 7mb green open .kibana_2 5fz9INfARTOnPe3ukAb-Ig 1 1 33 0 8.8mb 4.4mb green open logxx-xxxx-xxx-data-2021.01.22 irSvCD9cQ2uvBHkvpMSSEg 1 1 509953 0 344.7mb 172.3mb green open .kibana_1 EIUEMtDzSrePftYkAfTbEA 1 1 6 0 16.7kb 8.3kb green open logxx-xxxx-xxx-data-2021.01.14 wNB3aaBUSnKSQl_Be8kEsw 1 1 230341 0 114.4mb 57.2mb green open .tasks nXkgUyYrRn-NBJqFYoj5iA 1 1 1 0 13.9kb 6.9kb green open logxx-xxxx-xxx-data-2021.01.16 RSKmL8nLSCCgNa_PSM8j9w 1 1 84672 0 63.2mb 31.6mb green open logxx-xxxx-xxx-data-2021.01.15 sSEJDbZqR8yPe8n0grFbxw 1 1 223695 0 116.9mb 58.4mb green open .security-7 JpH0FVXsSJi5RNg-vnBszg 1 1 56 7 386.1kb 193kb green open logxx-xxxx-xxx-data-2021.01.18 DBern0wiRkmRnaGKpUuZ_g 1 1 670791 0 276.5mb 138.2mb green open .kibana-event-log-7.10.1 QiZQazsfS_e-3xx8tYW-nw 1 1 1 0 13.9kb 6.9kb green open .apm-custom-link N7ZXpvn4R6S5591BkS79qA 1 1 0 0 416b 208b green open .kibana_task_manager_1 FcDffMGUTJCesl34DGx_uw 1 1 6 3729 769.2kb 381.9kb green open logxx-xxxx-xxx-data-2021.01.19 19U6OoR0ShqMXb0xQyRgAQ 1 1 651965 0 304.9mb 152.4mb green open .async-search lantDqGURdq5AXXf4lRikA 1 1 9 9 11.6mb 5.8mb green open filebeat-7.10.1-2021.01.22-000001 1VRkhAA1R8ONXECuknyAxw 1 1 885123 0 626.4mb 318.5mb ``` ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 24, 2021, 11:46pm UTC](https://discuss.elastic.co/t/high-ram-usage-by-data-nodes-kubernetes/262008/2 "2021-01-24T23:46:34Z")
Welcome to our community! 😃 > [@itsron](#): > > I've noticed that the RAM usage is \>70% even when there are no queries. Why is that? It is possibly/probably the OS caching files that are being heavily used/written to in memory. ---
**Author:** ![itsron](https://avatars.discourse-cdn.com/v4/letter/i/a88e4f/32.png) [@itsron](https://discuss.elastic.co/u/itsron)\ **Post date:** [January 25, 2021, 3:44pm UTC](https://discuss.elastic.co/t/high-ram-usage-by-data-nodes-kubernetes/262008/3 "2021-01-25T15:44:41Z")
Thanks for your reply. I used `free -m -g` to see the memory usage: ``` total used free shared buff/cache available Mem: 15 9 0 0 5 5 Swap: 0 0 0 ``` which shows me that there is 5Gb of buff/cache and used+buff/cache roughly equal to what I see in `/cat_nodes` output. Do you think 16GB of RAM is not enough for the amount of data that I'm generating? Currently it is about 21M/day (~10-12GB of data) with 1 Primary/ 1 replica on a 4 data node cluster. I want to retain this data max 1 week. Sorry if I sound stupid, I'm new to elasticsearch. ---
**Author:** ![itsron](https://avatars.discourse-cdn.com/v4/letter/i/a88e4f/32.png) [@itsron](https://discuss.elastic.co/u/itsron)\ **Post date:** [January 26, 2021, 3:49pm UTC](https://discuss.elastic.co/t/high-ram-usage-by-data-nodes-kubernetes/262008/4 "2021-01-26T15:49:16Z")
@warkolm Also I can see even masters take 4.5Gb of RAM and has lots of page caches. Is this normal? I imagined masters shouldn't use so much resources. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 26, 2021, 7:58pm UTC](https://discuss.elastic.co/t/high-ram-usage-by-data-nodes-kubernetes/262008/5 "2021-01-26T19:58:32Z")
There's (basically) two memory "areas" when talking about use in Elasticsearch. There's the primary one which is heap, which is what you tell the JVM it's allowed to use and is what you can directly influence the size of. Then there's off-heap memory, which is mostly the OS managing commonly paged files to improve performance. You can manage this, but it's outside the scope of what we can help with. You really want to focus on your heap use, if you aren't getting heavy garbage collection, circuit breaker responses or OOM (which will be rare), or seeing slowness at all (which happens with a full heap) then you should be fine. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 23, 2021, 7:58pm UTC](https://discuss.elastic.co/t/high-ram-usage-by-data-nodes-kubernetes/262008/6 "2021-02-23T19:58:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.