We are running v7.10.1 on Kubernetes. We've 4 data nodes each with 4 cpus and 16gb RAM (8gb heap). Currently it has only metricbeat and filebeat writing data into it which is a total of ~1.5M docs/hour (~10GB/day,retained for 1 week). These nodes do not run anything else other than a couple of essential system pods (which do not consume any significant resources).
I've noticed that the RAM usage is >70% even when there are no queries. Why is that?
id name r m hc hp rc rp rm cpu dt du da load_1m load_5m
gb9i elasticsearch-data-0 cdhilrstw - 4.3gb 54 12.4gb 78 16gb 5 78.6gb 3.5gb 75gb 0.16 0.29
A8zB elasticsearch-data-1 cdhilrstw - 2.4gb 30 12.2gb 77 16gb 5 78.6gb 3.3gb 75.2gb 0.72 0.28
a-Lq elasticsearch-data-2 cdhilrstw - 2.3gb 29 9.9gb 62 16gb 3 78.6gb 1.1gb 77.4gb 0.16 0.14
Xa15 elasticsearch-data-3 cdhilrstw - 3.7gb 47 9.7gb 61 16gb 3 78.6gb 962.8mb 77.6gb 0.05 0.05
T3s2 elasticsearch-master-0 lm * 706.3mb 17 4.4gb 56 8gb 6 4.7gb 36.4mb 4.7gb 0.00 0.05
0ile elasticsearch-master-1 lm - 1.9gb 47 4.4gb 56 8gb 2 4.7gb 36.4mb 4.7gb 0.07 0.13
Cbi_ elasticsearch-master-2 lm - 186.7mb 4 4.4gb 56 8gb 3 4.7gb 36.4mb 4.7gb 0.10 0.08
e5E9 elasticsearch-master-3 lm - 1.4gb 35 4.4gb 55 8gb 3 4.7gb 36.5mb 4.7gb 0.48 0.37
_cat/shards
output:
index shard prirep state docs store node
.slm-history-3-000001 0 r STARTED elasticsearch-data-0
.slm-history-3-000001 0 p STARTED elasticsearch-data-2
.kibana_security_session_1 0 r STARTED elasticsearch-data-1
.kibana_security_session_1 0 p STARTED elasticsearch-data-2
ilm-history-3-000001 0 r STARTED elasticsearch-data-3
ilm-history-3-000001 0 p STARTED elasticsearch-data-1
.apm-agent-configuration 0 p STARTED 0 208b elasticsearch-data-0
.apm-agent-configuration 0 r STARTED 0 208b elasticsearch-data-2
.apm-custom-link 0 p STARTED 0 208b elasticsearch-data-3
.apm-custom-link 0 r STARTED 0 208b elasticsearch-data-1
.tasks 0 p STARTED 1 6.9kb elasticsearch-data-1
.tasks 0 r STARTED 1 6.9kb elasticsearch-data-2
.kibana-event-log-7.10.1 0 p STARTED 1 6.9kb elasticsearch-data-0
.kibana-event-log-7.10.1 0 r STARTED 1 6.9kb elasticsearch-data-3
.kibana_task_manager_1 0 r STARTED 6 367.7kb elasticsearch-data-0
.kibana_task_manager_1 0 p STARTED 6 362.2kb elasticsearch-data-1
.kibana_1 0 p STARTED 6 8.3kb elasticsearch-data-3
.kibana_1 0 r STARTED 6 8.3kb elasticsearch-data-1
.async-search 0 p STARTED 9 5.8mb elasticsearch-data-0
.async-search 0 r STARTED 9 5.8mb elasticsearch-data-2
.kibana_2 0 r STARTED 32 4.4mb elasticsearch-data-3
.kibana_2 0 p STARTED 32 2.8mb elasticsearch-data-2
.security-7 0 r STARTED 56 193kb elasticsearch-data-0
.security-7 0 p STARTED 56 193kb elasticsearch-data-1
logxx-xxxx-xxx-data-2021.01.23 0 p STARTED 16993 7mb elasticsearch-data-0
logxx-xxxx-xxx-data-2021.01.23 0 r STARTED 16993 7mb elasticsearch-data-2
metricbeat-8.0.0-2021.01.23-000001 0 p STARTED 26791 14.8mb elasticsearch-data-0
metricbeat-8.0.0-2021.01.23-000001 0 r STARTED 26791 14.9mb elasticsearch-data-3
logxx-xxxx-xxx-data-2021.01.16 0 r STARTED 84672 31.6mb elasticsearch-data-1
logxx-xxxx-xxx-data-2021.01.16 0 p STARTED 84672 31.6mb elasticsearch-data-2
logxx-xxxx-xxx-data-2021.01.21 0 p STARTED 84960 24.8mb elasticsearch-data-3
logxx-xxxx-xxx-data-2021.01.21 0 r STARTED 84960 24.8mb elasticsearch-data-2
logxx-xxxx-xxx-data-2021.01.15 0 r STARTED 223695 58.4mb elasticsearch-data-0
logxx-xxxx-xxx-data-2021.01.15 0 p STARTED 223695 58.4mb elasticsearch-data-1
logxx-xxxx-xxx-data-2021.01.14 0 r STARTED 230341 57.2mb elasticsearch-data-3
logxx-xxxx-xxx-data-2021.01.14 0 p STARTED 230341 57.2mb elasticsearch-data-1
logxx-xxxx-xxx-data-2021.01.22 0 p STARTED 509953 172.3mb elasticsearch-data-0
logxx-xxxx-xxx-data-2021.01.22 0 r STARTED 509953 172.3mb elasticsearch-data-3
logxx-xxxx-xxx-data-2021.01.19 0 p STARTED 651965 152.4mb elasticsearch-data-0
logxx-xxxx-xxx-data-2021.01.19 0 r STARTED 651965 152.4mb elasticsearch-data-2
logxx-xxxx-xxx-data-2021.01.18 0 r STARTED 670791 138.2mb elasticsearch-data-3
logxx-xxxx-xxx-data-2021.01.18 0 p STARTED 670791 138.2mb elasticsearch-data-2
logxx-xxxx-xxx-data-2021.01.20 0 r STARTED 765555 159.7mb elasticsearch-data-3
logxx-xxxx-xxx-data-2021.01.20 0 p STARTED 765555 159.7mb elasticsearch-data-1
filebeat-7.10.1-2021.01.22-000001 0 r STARTED 881964 292.5mb elasticsearch-data-3
filebeat-7.10.1-2021.01.22-000001 0 p STARTED 882031 317.3mb elasticsearch-data-2
metricbeat-7.10.1-2021.01.22-000001 0 p STARTED 7443102 2.7gb elasticsearch-data-1
metricbeat-7.10.1-2021.01.22-000001 0 r STARTED 7443119 2.6gb elasticsearch-data-0
_cat/indices
output:
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open metricbeat-7.10.1-2021.01.22-000001 c0QgKkR0RKyLyALbO4gHJA 1 1 7476803 0 5.4gb 2.7gb
green open metricbeat-8.0.0-2021.01.23-000001 _ey3bMu1Qt6uGrw1MkR_aQ 1 1 26791 0 29.7mb 14.8mb
green open .apm-agent-configuration xm1KHwtOTrWZa4pIGtiwaQ 1 1 0 0 416b 208b
green open logxx-xxxx-xxx-data-2021.01.21 -4VTtMV-RG26QmDgBr4Gyg 1 1 84960 0 49.6mb 24.8mb
green open logxx-xxxx-xxx-data-2021.01.20 v4us1dBZRe69o7yxPONmRA 1 1 765555 0 319.5mb 159.7mb
green open logxx-xxxx-xxx-data-2021.01.23 4-KqF7EhSfGe0aGNIyhckw 1 1 16993 0 14.1mb 7mb
green open .kibana_2 5fz9INfARTOnPe3ukAb-Ig 1 1 33 0 8.8mb 4.4mb
green open logxx-xxxx-xxx-data-2021.01.22 irSvCD9cQ2uvBHkvpMSSEg 1 1 509953 0 344.7mb 172.3mb
green open .kibana_1 EIUEMtDzSrePftYkAfTbEA 1 1 6 0 16.7kb 8.3kb
green open logxx-xxxx-xxx-data-2021.01.14 wNB3aaBUSnKSQl_Be8kEsw 1 1 230341 0 114.4mb 57.2mb
green open .tasks nXkgUyYrRn-NBJqFYoj5iA 1 1 1 0 13.9kb 6.9kb
green open logxx-xxxx-xxx-data-2021.01.16 RSKmL8nLSCCgNa_PSM8j9w 1 1 84672 0 63.2mb 31.6mb
green open logxx-xxxx-xxx-data-2021.01.15 sSEJDbZqR8yPe8n0grFbxw 1 1 223695 0 116.9mb 58.4mb
green open .security-7 JpH0FVXsSJi5RNg-vnBszg 1 1 56 7 386.1kb 193kb
green open logxx-xxxx-xxx-data-2021.01.18 DBern0wiRkmRnaGKpUuZ_g 1 1 670791 0 276.5mb 138.2mb
green open .kibana-event-log-7.10.1 QiZQazsfS_e-3xx8tYW-nw 1 1 1 0 13.9kb 6.9kb
green open .apm-custom-link N7ZXpvn4R6S5591BkS79qA 1 1 0 0 416b 208b
green open .kibana_task_manager_1 FcDffMGUTJCesl34DGx_uw 1 1 6 3729 769.2kb 381.9kb
green open logxx-xxxx-xxx-data-2021.01.19 19U6OoR0ShqMXb0xQyRgAQ 1 1 651965 0 304.9mb 152.4mb
green open .async-search lantDqGURdq5AXXf4lRikA 1 1 9 9 11.6mb 5.8mb
green open filebeat-7.10.1-2021.01.22-000001 1VRkhAA1R8ONXECuknyAxw 1 1 885123 0 626.4mb 318.5mb