Elasticsearch Disk Space Issue

bagafoot · October 21, 2024, 8:46am

Hello I have an elk and strugling with management disk space.
"The indices on this server, with only 5 pages in total, do not even amount to 100GB, but the Elasticsearch data directory shows 1.4TB of usage. What is consuming this space, and how can I manage it to prevent this from happening?"

indices:

elk version
{
"name": "test-node01",
"cluster_name": "test",
"cluster_uuid": "WCWI6ipYQY2n3pA_zFY2zQ",
"version": {
"number": "8.6.0",
"build_flavor": "default",
"build_type": "deb",
"build_hash": "f67ef2df40237445caa70e2fef79471cc608d70d",
"build_date": "2023-01-04T09:35:21.782467981Z",
"build_snapshot": false,
"lucene_version": "9.4.2",
"minimum_wire_compatibility_version": "7.17.0",
"minimum_index_compatibility_version": "7.0.0"
},
"tagline": "You Know, for Search"
}

dadoonet · October 21, 2024, 10:31am

Few comments:

What is the use case for your data?
It seems that you are asking for 1+ replicas but have only onde node. Is that intended?
What is the output of GET _cat/nodes?v
You should upgrade to 8.15.3, specifically if you are doing logs or metrics.
If you are doing a lot of updates and/or deletes, running a forcemerge call might help to reclaim some disk space.
What is your mapping? If you are using the default mapping, that might explain a lot as text data is stored, indexed, computed multiple times (text vs keyword)

The first answers will help to understand a bit more...

bagafoot · October 21, 2024, 1:16pm

Using it for dwh, creating and searching indexes.
No, it is not intended

ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role   	master name
x.x.x.x       50       	   86  		   41   5.43    6.43     6.76 	cdfhilmrstw *   test-node01

--
Already using merge, meybe twice a week recreating indexes.
Yes, there is default maping

dadoonet · October 21, 2024, 1:37pm

So reduce the number of replicas. That won't change anything but this much more consistent with the architecture you defined.
Change the mapping to avoid things you don't need. Specifically look at the text fields.
May be don't store the _source field if you don't need it. Have also a look at _source field | Elasticsearch Guide [8.15] | Elastic

Christian_Dahlqvist · October 21, 2024, 1:40pm

I would recommend you read and follow the guidelines in these docs.

bagafoot · October 21, 2024, 2:55pm

thanks a lot @dadoonet and @Christian_Dahlqvist for your helps

bagafoot · October 22, 2024, 7:12am

last question, replicas means replicate index and multiply index.
example, in screenshot there is 3.8Gb size index with 4 replicas, that means 3.8x4 gb or just 3.8gb

leandrojmp · October 22, 2024, 12:48pm

The value in the Kibana UI takes replicas into account, it means that if you have 4 shards and 4 replicas and it says that you are using 3.8 GB, than this is the total space used by both primaries and replicas.

Also, you just shared the Index page, you also need to check the data stream.

You are on versiion 8, if you are using any beat on version 8 or Elastic Agent, they will store data on data streama, that will show up in the data stream tab, not in the index page.

A quick way to know how much space your indices are consuming is using the GET _cat/indices?v

Please run, this in Kibana Dev Tools and share the result.

bagafoot · October 22, 2024, 2:42pm

in documantation disable _source field not recommended.

I couldnt disable _source though

some index names are masked,

index                                                             health status rep store.size docs.count
k*******************s                                             green  open     0     37.8gb   35772078
k*************n                                                   green  open     0      8.5gb   14969117
.ds-metricbeat-8.6.0-2024.08.30-000012                            green  open     0      4.8gb    4327706
.ds-metricbeat-8.6.0-2024.05.02-000008                            green  open     0      4.8gb    4303695
.ds-metricbeat-8.6.0-2024.06.01-000009                            green  open     0      4.8gb    4309461
.ds-metricbeat-8.6.0-2024.07.31-000011                            green  open     0      4.7gb    4329049
.ds-metricbeat-8.6.0-2024.07.01-000010                            green  open     0      4.7gb    4326789
.ds-metricbeat-8.6.0-2024.04.02-000007                            green  open     0      4.7gb    4301731
.ds-.monitoring-es-8-mb-2023.04.24-000029                         green  open     0      3.8gb    6433931
d*****************a                                               green  open     0      3.8gb   11417168
.ds-.monitoring-es-8-mb-2023.04.21-000028                         green  open     0      3.7gb    6321096
.ds-.monitoring-es-8-mb-2023.04.18-000027                         green  open     0      3.7gb    6266475
.ds-metricbeat-8.6.0-2024.09.29-000013                            green  open     0      3.7gb    3276701
.ds-.monitoring-es-8-mb-2023.04.15-000026                         green  open     0      3.6gb    6214489
d****************a                                                green  open     0      3.6gb   21593548
.ds-.monitoring-es-8-mb-2023.04.12-000025                         green  open     0      3.6gb    6164313
.ds-metricbeat-8.6.0-2023.03.26-000003                            green  open     0      3.6gb    4234140
.ds-.monitoring-es-8-mb-2023.04.09-000024                         green  open     0      3.6gb    6110778
.ds-metricbeat-8.6.0-2023.01.25-000001                            green  open     0      3.6gb    4562304
.ds-metricbeat-8.6.0-2023.02.24-000002                            green  open     0      3.6gb    4200400
.ds-.monitoring-es-8-mb-2023.04.06-000023                         green  open     0      3.5gb    6042744
.ds-.monitoring-es-8-mb-2023.04.03-000022                         green  open     0      3.5gb    5980561
.ds-.monitoring-es-8-mb-2023.03.31-000021                         green  open     0      3.4gb    5932562
.ds-.monitoring-es-8-mb-2023.03.28-000020                         green  open     0      3.4gb    5877087
.ds-metricbeat-8.6.0-2023.04.25-000004                            green  open     0      3.4gb    4073262
.ds-.monitoring-es-8-mb-2023.03.25-000019                         green  open     0      3.4gb    5810528
k****************i                                                green  open     0        3gb    7068495
.ds-metricbeat-8.6.0-2023.05.25-000005                            green  open     0        3gb    3535579
.ds-.monitoring-es-8-mb-2023.03.22-000018                         green  open     0        3gb    5131127
.ds-.monitoring-es-8-mb-2023.03.19-000017                         green  open     0      2.8gb    4863419
.ds-.monitoring-es-8-mb-2024.10.21-000116                         green  open     0      2.8gb    4268892
.ds-.monitoring-es-8-mb-2023.03.16-000016                         green  open     0      2.8gb    4752196
.ds-.monitoring-es-8-mb-2023.03.13-000015                         green  open     0      2.6gb    4407949
.ds-.monitoring-es-8-mb-2023.05.15-000036                         green  open     0      2.6gb    4704344
.ds-.monitoring-es-8-mb-2023.03.10-000014                         green  open     0      2.5gb    4136776
.ds-.monitoring-es-8-mb-2023.06.17-000047                         green  open     0      2.4gb    4376882
.ds-.monitoring-es-8-mb-2023.03.07-000013                         green  open     0      2.4gb    3964123
.ds-.monitoring-es-8-mb-2023.03.04-000012                         green  open     0      2.4gb    3876040
.ds-.monitoring-es-8-mb-2023.03.01-000011                         green  open     0      2.3gb    3675043
.ds-.monitoring-es-8-mb-2023.02.26-000010                         green  open     0      2.2gb    3537521
.ds-.monitoring-es-8-mb-2023.02.23-000009                         green  open     0      2.2gb    3463723
k*************a                                                   green  open     0      1.7gb    6362287
.ds-.monitoring-es-8-mb-2023.02.20-000008                         green  open     0      1.5gb    2453415
d**************************i                                      green  open     0      1.4gb   14656589
.ds-.monitoring-es-8-mb-2023.02.17-000007                         green  open     0      1.2gb    2060348
.ds-.monitoring-es-8-mb-2023.02.14-000006                         green  open     0      1.1gb    1814772
k*******************a                                             green  open     0      1.1gb    1707607
.ds-.monitoring-es-8-mb-2023.02.11-000005                         green  open     0      1.1gb    1731479
.ds-.monitoring-es-8-mb-2023.02.08-000004                         green  open     0        1gb    1692706
t*******r                                                         green  open     0    995.6mb    2518567
.ds-.monitoring-es-8-mb-2023.02.05-000003                         green  open     0    944.3mb    1346711
.ds-.monitoring-es-8-mb-2023.02.02-000002                         green  open     0    826.3mb    1135410
k*************i                                                   green  open     0      688mb    1185339
.ds-.monitoring-es-8-mb-2023.01.30-000001                         green  open     0    670.7mb     897106
d**********************                                           green  open     0    647.5mb   14785723
k**********************                                           green  open     0    358.3mb     881026
k**********************                                           green  open     0    320.3mb     478985
k**********************                                           green  open     0    169.4mb        973
t**********************                                           green  open     0    153.2mb     660000
k**********************                                           yellow open     1    153.2mb      91185
k**********************                                           yellow open     1    139.1mb      78139
.**********************-default-2023.02.22-000001                 green  open     0    125.1mb     246285
k******************************j                                  green  open     0     92.8mb    1843461
k******************************                                   green  open     0     79.7mb     294067
d******************************                                   green  open     0     78.5mb     889016
k******************************                                   green  open     0     67.2mb         81
k******************************                                   green  open     0       45mb      71704
p******************************                                   green  open     0     29.4mb     294049
.ds-logs-system.syslog-default-2023.02.22-000001                  green  open     0      8.7mb     138252
k******************o                                              green  open     0      8.5mb      89707
.ds-metrics-system.diskio-default-2023.02.22-000001               green  open     0      5.2mb      33023
.ds-metrics-system.process-default-2023.02.22-000001              green  open     0        5mb       6238
k************                                                     green  open     0        4mb        319
s************i                                                    green  open     0      3.8mb      13293
.monitoring-kibana-7-2024.10.21                                   green  open     0      3.5mb      11498
.monitoring-kibana-7-2024.10.18                                   green  open     0      3.5mb      17274
.monitoring-kibana-7-2024.10.17                                   green  open     0      3.5mb      17278
.monitoring-kibana-7-2024.10.19                                   green  open     0      3.5mb      17280
.monitoring-kibana-7-2024.10.16                                   green  open     0      3.3mb      17280
.monitoring-kibana-7-2024.10.22                                   green  open     0      2.8mb       8634
.ds-.monitoring-es-8-mb-2023.06.20-000048                         green  open     0        2mb       3496
k************************************y                            green  open     0      1.8mb      16020
.ds-metrics-system.network-default-2023.02.22-000001              green  open     0      1.7mb       7556
k********************************2                                green  open     0      1.7mb       2787
.ds-metricbeat-8.6.0-2023.06.24-000006                            green  open     0      1.1mb        138
.ds-logs-system.auth-default-2023.02.22-000001                    green  open     0    980.7kb      10760
k*********************************                                green  open     0    885.3kb       1394
k*********************************                                green  open     0    825.6kb       1477
.ds-metrics-elastic_agent.elastic_agent-default-2023.02.22-000001 green  open     0    744.4kb       2064
.ds-metrics-elastic_agent.metricbeat-default-2023.02.22-000001    green  open     0    657.1kb       1374
.ds-metrics-elastic_agent.filebeat-default-2023.02.22-000001      green  open     0    565.1kb       1376
.ds-metrics-system.filesystem-default-2023.02.22-000001           green  open     0    384.4kb        920
.ds-logs-elastic_agent.metricbeat-default-2023.02.22-000001       green  open     0      381kb        809
.ds-metrics-system.process.summary-default-2023.02.22-000001      green  open     0    363.6kb        685
.ds-metrics-system.cpu-default-2023.02.22-000001                  green  open     0    305.1kb        687
.ds-metrics-system.memory-default-2023.02.22-000001               green  open     0    288.8kb        686
.ds-metrics-system.socket_summary-default-2023.02.22-000001       green  open     0      277kb        687
.ds-logs-elastic_agent.filebeat-default-2023.02.22-000001         green  open     0    271.3kb        545
k**************************                                       green  open     0    269.8kb        499
k**************************                                       green  open     0    267.4kb        499
.ds-metrics-system.load-default-2023.02.22-000001                 green  open     0      245kb        688
m*************************y                                       green  open     0    230.8kb        100
.ds-metrics-system.uptime-default-2023.02.22-000001               green  open     0    210.2kb        687
.ds-metrics-system.fsstat-default-2023.02.22-000001               green  open     0    155.4kb        115
.ds-logs-elastic_agent-default-2023.02.22-000001                  green  open     0     95.5kb         45
r*******************                                              green  open     0     11.7kb          4
r*******************                                              green  open     0      8.7kb          1
y*******************                                              green  open     0      8.7kb          1
g*******************                                              green  open     0      6.1kb          1
d*******************                                              green  open     0      4.6kb          1
.ds-logs-postgresql.log-default-2023.03.24-000002                 green  open     0       225b          0
.ds-logs-system.syslog-default-2023.03.24-000002                  green  open     0       225b          0
.ds-metrics-elastic_agent.metricbeat-default-2023.03.24-000002    green  open     0       225b          0
.monitoring-kibana-7-2024.10.20                                   green  open     0       225b          0
i**************                                                   green  open     0       225b          0
.ds-metrics-system.network-default-2023.03.24-000002              green  open     0       225b          0
0                                                                 green  open     0       225b          0
.ds-logs-elastic_agent-default-2023.03.24-000002                  green  open     0       225b          0
.ds-metrics-elastic_agent.elastic_agent-default-2023.03.24-000002 green  open     0       225b          0
.ds-metrics-system.cpu-default-2023.03.24-000002                  green  open     0       225b          0
.ds-logs-elastic_agent.metricbeat-default-2023.03.24-000002       green  open     0       225b          0
.ds-metrics-system.fsstat-default-2023.03.24-000002               green  open     0       225b          0
metrics-endpoint.metadata_current_default                         green  open     0       225b          0
asd                                                               green  open     0       225b          0
.ds-metrics-system.diskio-default-2023.03.24-000002               green  open     0       225b          0
.items-default-000001                                             green  open     0       225b          0
.ds-metrics-system.uptime-default-2023.03.24-000002               green  open     0       225b          0
.ds-metrics-system.socket_summary-default-2023.03.24-000002       green  open     0       225b          0
t**************                                                   green  open     0       225b          0
test                                                              green  open     0       225b          0
.ds-logs-system.auth-default-2023.03.24-000002                    green  open     0       225b          0
.ds-metrics-elastic_agent.filebeat-default-2023.03.24-000002      green  open     0       225b          0
.ds-metrics-system.memory-default-2023.03.24-000002               green  open     0       225b          0
.ds-metrics-system.filesystem-default-2023.03.24-000002           green  open     0       225b          0
.ds-metrics-system.process.summary-default-2023.03.24-000002      green  open     0       225b          0
.lists-default-000001                                             green  open     0       225b          0
g***************                                                  green  open     0       225b          0
.ds-metrics-system.load-default-2023.03.24-000002                 green  open     0       225b          0
.ds-metrics-system.process-default-2023.03.24-000002              green  open     0       225b          0
.ds-logs-elastic_agent.filebeat-default-2023.03.24-000002         green  open     0       225b          0

leandrojmp · October 22, 2024, 3:25pm

Do not disable _source this breaks a lot of things, for example if you use Kibana you won't be able to see your data anymore in discover or dashboards.

These are all your indices? You have some datastreams where the backing indices sums to something around 130 GB, but nothing closer to the 685 GB that you shared before.

Is it possible that you have other files in the same partition taking up some space?

Please run ls -larth on the /repo path and on the /elastic path and share the result.

Also, if possible run du -h --max-depth=2 on the /repo path.

The path name being /repo suggests that this is being used for snapshots, can you confirm that? What is the result of running GET /_cat/snapshots in Kibana Dev Tools?

Please also share your elasticsearch.yml.

bagafoot · October 23, 2024, 8:00am

I found some hidden indexes, then pages rise up to 17

there are data streams sum up to 130 gb

GET /_cat/allocation?v&h=disk.used

disk.used
    894gb

du -h --max-depth=2 repo

image470×229 7.03 KB

GET /_cat/snapshots

weekly-snap-2024.10.07-dx7pr-fiqeae1vop9hy-aq csb-elastic-backup SUCCESS 1728691199 23:59:59 1728691749 00:09:09 9.1m 232 296 0 296
weekly-snap-2024.10.14-knoqx3lotumksolhwuqrbq csb-elastic-backup SUCCESS 1729295999 23:59:59 1729296840 00:14:00  14m 243 319 0 319

elasticsearch.yml

cluster.name: test
node.name: test-node01
path.repo: /elastic/repo
path.data: /elastic
path.logs: /var/log/elasticsearch
network.host: x.x.x.x
http.port: 9200
discovery.seed_hosts: ["x.x.x.x"]
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.client_authentication: optional
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
cluster.initial_master_nodes: ["x.x.x.x"]
http.host: 0.0.0.0
transport.host: 0.0.0.0

leandrojmp · October 23, 2024, 12:27pm

You have snapshots on the same disk, this is what is taking too much space, not your indices.

As you could see your indices use something close to 150 GB, all the rest inside the /repo paht are snapshots.

Normally you should use different data paths for elasticsearch indices and snapshots since you may want to keep snapshots for a long time and can use a cheaper storage.

File System snapshots would be normally saved on network shares.

Did GET _/cat/snapshots only returned 2 snapshots or you cut some lines?

You can go into Snapshot & Restore in Index Management to see all your snapshots.

It is not your indices that are taking up space, it is your snapshots, so to free space you would need to remove some snapshots, but this means deleting old data.

If you don't want to keep it, then you can delete.

bagafoot · October 23, 2024, 1:26pm

nope, there are two snapshots, I could find size this snaps. I will drop oldest one.
thanks a lot for your help.