# Storage footprint not the same as cummulative index size, why is that, **URL:** \ **Category:** Elasticsearch\ **Created:** [November 30, 2018, 9:17am UTC](https://discuss.elastic.co/t/storage-footprint-not-the-same-as-cummulative-index-size-why-is-that/158872 "2018-11-30T09:17:32Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![jacobot](https://avatars.discourse-cdn.com/v4/letter/j/4491bb/32.png) [@jacobot](https://discuss.elastic.co/u/jacobot)\ **Post date:** [November 30, 2018, 9:17am UTC](https://discuss.elastic.co/t/storage-footprint-not-the-same-as-cummulative-index-size-why-is-that/158872/1 "2018-11-30T09:17:32Z")
Hi, i am issueing this query `GET /_cat/allocation?v&h=disk.indices,disk.avail,disk.used,disk.total,disk.percent` response ```auto disk.indices disk.avail disk.used disk.total disk.percent 10.2gb 15.2gb 34.6gb 49.9gb 69 ``` So indices is 10.2gb and disk used is 34.6gb?? where is the difference comming from looking at the `data` directory its indeed 35Gb ```auto du -h -d 5 44K ./elasticsearch/nodes/0/_state 184K ./elasticsearch/nodes/0/indices/dBvqitmZRt-Fmuc0m27ngQ 695M ./elasticsearch/nodes/0/indices/LT-pbZUmQUWTVj_eZ6ueoQ 211M ./elasticsearch/nodes/0/indices/UrkzhOUPT2mGq5CIxjE2jw 2.2M ./elasticsearch/nodes/0/indices/aNoeXSc1R5-Yv5mi5kAEVw 2.0M ./elasticsearch/nodes/0/indices/FTyJMumRRQ6KlDMkJPpg0g 1.4M ./elasticsearch/nodes/0/indices/FaHslxLmRPmt_epBjRl4oA 84K ./elasticsearch/nodes/0/indices/sZrzFZaUQCq2lntAfOVG1A 6.9M ./elasticsearch/nodes/0/indices/OMgjPF0kSdacpMlDFbhV9A 214M ./elasticsearch/nodes/0/indices/l9nus3Q_SKu5G6lFHsmshw 9.8M ./elasticsearch/nodes/0/indices/bh-yRTBQSOWejk2JaPGGQw 1.9G ./elasticsearch/nodes/0/indices/mxEqic8MTw2W1sDbDsWwYQ 2.0G ./elasticsearch/nodes/0/indices/aw9QKXJbRMinedvKUHeAmw 2.2M ./elasticsearch/nodes/0/indices/L7pYWKHjQVaO8R21hi_AmA 328M ./elasticsearch/nodes/0/indices/NClre8MZSomW4-Hxab3aow 13M ./elasticsearch/nodes/0/indices/7mc_Q_ziTvelFrAX-d_umg 3.0M ./elasticsearch/nodes/0/indices/PN5v9z7dSoip7uMXwSglqg 812K ./elasticsearch/nodes/0/indices/cwesDBMVTOOIYD50690Rhw 186M ./elasticsearch/nodes/0/indices/r1G0zHTJRTaEUMSrjcwj8Q 1.9G ./elasticsearch/nodes/0/indices/yk0y91gNQZOdhMKftSVa1g 1.9G ./elasticsearch/nodes/0/indices/wfu1bw9oR3W3iDZS8x_6OQ 1.9G ./elasticsearch/nodes/0/indices/x_qsGMdqQZi8Es_KKvWuCA 1.9G ./elasticsearch/nodes/0/indices/-CI6VqaYR8KMaHyesiDrdA 2.2M ./elasticsearch/nodes/0/indices/MwUf5GHcSxmmNwRr80szLw 5.6M ./elasticsearch/nodes/0/indices/9efquwiUTOu0Uyba2m3wIQ 1.9G ./elasticsearch/nodes/0/indices/jZdwR2BuSfWK_wR0k_Jx8w 1.9G ./elasticsearch/nodes/0/indices/K5aLBsDjRPGWEuK3AdWgVA 1.9G ./elasticsearch/nodes/0/indices/rqzC5P0KQqeVlPOiaMRJ-g 226M ./elasticsearch/nodes/0/indices/EgLHTHW0S6i6eJT1HOJi7w 1.9G ./elasticsearch/nodes/0/indices/oNRlGK97S7iewBzr2o0dyA 1.9G ./elasticsearch/nodes/0/indices/LBWkHnAeTvaBSSOldBO8Tg 156M ./elasticsearch/nodes/0/indices/MXiwXUudSY2yY5cm27mFcQ 5.9M ./elasticsearch/nodes/0/indices/YhcziY5OTLiLyl6pTAV0WA 310M ./elasticsearch/nodes/0/indices/SFbJg2oYRQCPGbdsdb2LZQ 303M ./elasticsearch/nodes/0/indices/4r3_DX-NTzaK-6IieDkEFA 1.9M ./elasticsearch/nodes/0/indices/8l4-SA5PR6-4MxGXWG7fgA 193M ./elasticsearch/nodes/0/indices/XmiRCtGKRuqIWtLx2J_sbA 25M ./elasticsearch/nodes/0/indices/z2R2msq_QaS6OGF9JBKU7w 1.9G ./elasticsearch/nodes/0/indices/2A6_bEZ6RKe16dkXT2Xa_Q 1.9G ./elasticsearch/nodes/0/indices/Fpj3EBzYRke83n--8Jz0ow 1.9G ./elasticsearch/nodes/0/indices/LsrLw3n2QoKTcxBwQ7A_Bg 285M ./elasticsearch/nodes/0/indices/zWlP4T0gQ5KvOc9Q0zQkkw 1.9G ./elasticsearch/nodes/0/indices/y15InfVUTEWNW_pj4EjW7w 1.9G ./elasticsearch/nodes/0/indices/1JLAYcPiRGO8zaE8v_1iIw 207M ./elasticsearch/nodes/0/indices/P4BK8a0ERwy3Rvn4vf2K4Q 1.9G ./elasticsearch/nodes/0/indices/9sV3vtiNRhKI05YC1VIwoQ 5.0M ./elasticsearch/nodes/0/indices/6uPdPVNeTBGbvdxp0Cc0XQ 35G ./elasticsearch/nodes/0/indices 35G ./elasticsearch/nodes/0 35G ./elasticsearch/nodes 35G ./elasticsearch 0 ./logstash 35G . ``` why is there a difference? Note, we did dropped a lot of indices recently, are these somehow lingering files?? Cheers, ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 30, 2018, 10:01am UTC](https://discuss.elastic.co/t/storage-footprint-not-the-same-as-cummulative-index-size-why-is-that/158872/2 "2018-11-30T10:01:53Z")
Can this be caused by the transaction log? What is your version? ---
**Author:** ![jacobot](https://avatars.discourse-cdn.com/v4/letter/j/4491bb/32.png) [@jacobot](https://discuss.elastic.co/u/jacobot)\ **Post date:** [November 30, 2018, 12:43pm UTC](https://discuss.elastic.co/t/storage-footprint-not-the-same-as-cummulative-index-size-why-is-that/158872/3 "2018-11-30T12:43:52Z")
version info, how do i find out that the translog are the issue and will it be removed under space pressure? ```auto GET / { "name" : "43aC6EG", "cluster_name" : "eds-cluster-elk", "cluster_uuid" : "z61nh_jpRoaGaYQpPFSJLg", "version" : { "number" : "6.4.2", "build_flavor" : "default", "build_type" : "tar", "build_hash" : "04711c2", "build_date" : "2018-09-26T13:34:09.098244Z", "build_snapshot" : false, "lucene_version" : "7.4.0", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" } ``` Not sure how to fetch transaction log size how to find it here are our cluster settings ```auto { "persistent": { "action": { "auto_create_index": "+.kibana*, +.monitoring*, +.watcher*, +.prod_eds_messages*, +.dev_eds_messages*" }, "indices": { "recovery": { "max_bytes_per_sec": "50mb" } }, "xpack": { "monitoring": { "collection": { "enabled": "true" } } } }, "transient": {} } ``` ---
**Author:** ![jacobot](https://avatars.discourse-cdn.com/v4/letter/j/4491bb/32.png) [@jacobot](https://discuss.elastic.co/u/jacobot)\ **Post date:** [November 30, 2018, 4:23pm UTC](https://discuss.elastic.co/t/storage-footprint-not-the-same-as-cummulative-index-size-why-is-that/158872/4 "2018-11-30T16:23:46Z")
Hi, we are reaching, 85% threshold disk usage soon, (( ```auto GET /_cat/allocation?v&h=disk.indices,disk.avail,disk.used,disk.total,disk.percent disk.indices disk.avail disk.used disk.total disk.percent 15.8gb 8.1gb 41.8gb 49.9gb 83 ``` 15.8G of data, total disk 49.9gb, disk used 41.8gb, is it going to flush soon (i hope?) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 30, 2018, 4:42pm UTC](https://discuss.elastic.co/t/storage-footprint-not-the-same-as-cummulative-index-size-why-is-that/158872/5 "2018-11-30T16:42:05Z")
May be you can try to run a flush and see if it helps? [https://www.elastic.co/guide/en/elasticsearch/reference/6.5/indices-flush.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/indices-flush.html) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 30, 2018, 4:50pm UTC](https://discuss.elastic.co/t/storage-footprint-not-the-same-as-cummulative-index-size-why-is-that/158872/6 "2018-11-30T16:50:01Z")
In recent versions the translog is kept around for a while as it allows recovery based on sequence numbers which can speed up recovery significantly. [You can tune this](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/index-modules-translog.html#_translog_settings) if you want to. ---
**Author:** ![jacobot](https://avatars.discourse-cdn.com/v4/letter/j/4491bb/32.png) [@jacobot](https://discuss.elastic.co/u/jacobot)\ **Post date:** [November 30, 2018, 4:51pm UTC](https://discuss.elastic.co/t/storage-footprint-not-the-same-as-cummulative-index-size-why-is-that/158872/7 "2018-11-30T16:51:39Z")
I did `POST _all/_flush?wait_if_ongoing=true&force=true` reponse ```auto { "_shards": { "total": 388, "successful": 208, "failed": 0 } } ``` nothing changed ```auto disk.indices disk.avail disk.used disk.total disk.percent 16.3gb 7.9gb 42gb 49.9gb 84 ``` ---
**Author:** ![jacobot](https://avatars.discourse-cdn.com/v4/letter/j/4491bb/32.png) [@jacobot](https://discuss.elastic.co/u/jacobot)\ **Post date:** [November 30, 2018, 5:01pm UTC](https://discuss.elastic.co/t/storage-footprint-not-the-same-as-cummulative-index-size-why-is-that/158872/8 "2018-11-30T17:01:23Z")
we do have high insert rate (bulk insert) Sorry i dont find the doc perticularly clear _`index.translog.retention.size`_ _The total size of translog files to keep. Keeping more translog files increases the chance of performing an operation based sync when recovering replicas. If the translog files are not sufficient, replica recovery will fall back to a file based sync. Defaults to `512mb`_ _`index.translog.retention.age`_ _The maximum duration for which translog files will be kept. Defaults to `12h` ._ I must change these params to keep translog disk footprint to a certain size???? We do `_bulk` api inserts of 500-1500/sec ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/3/b3a2805dda7df7d36bd7916eb0f368b33602a007.png) **Note: its not critical data at the moment, its still an mvp at the moment, so its no problem to change params and experiment** ---
**Author:** ![jacobot](https://avatars.discourse-cdn.com/v4/letter/j/4491bb/32.png) [@jacobot](https://discuss.elastic.co/u/jacobot)\ **Post date:** [November 30, 2018, 5:56pm UTC](https://discuss.elastic.co/t/storage-footprint-not-the-same-as-cummulative-index-size-why-is-that/158872/9 "2018-11-30T17:56:14Z")
its is flushing and keeping it below 85% nice, ```auto disk.indices disk.avail disk.used disk.total disk.percent 16.9gb 8.6gb 41.3gb 49.9gb 82 ``` thanks for your help, it seems elastic will cleanup as needed ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 30, 2018, 6:14pm UTC](https://discuss.elastic.co/t/storage-footprint-not-the-same-as-cummulative-index-size-why-is-that/158872/10 "2018-11-30T18:14:12Z")
If your index is not "stable" because you are doing a lot of indexing, then many things can happen. For example segment merge. Which means that at some point of time you might have 2 segments containing each let say 100k docs. Elasticsearch will create a new segment containing 200k docs and then once done will remove the 2 old segments. Which means that during this operation you might use more disk space than needed. My 2 cents. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 1, 2018, 8:19am UTC](https://discuss.elastic.co/t/storage-footprint-not-the-same-as-cummulative-index-size-why-is-that/158872/11 "2018-12-01T08:19:33Z")
> [@dadoonet](#): > > Can this be caused by the transaction log? Yes, the `disk.indices` statistic only counts the store, i.e. segment files, and not the translog. Here is a cluster containing only one shard with ~1MB of segment data and ~4MB of translog: ```plaintext $ curl 'http://localhost:9200/_cat/allocation?v&h=disk.indices' disk.indices 1015.7kb $ du -k elasticsearch-6.5.1/data-0 ... 4120 elasticsearch-6.5.1/data-0/nodes/0/indices/Pw0fY116RE2EuySrE1zLyQ/0/translog 1064 elasticsearch-6.5.1/data-0/nodes/0/indices/Pw0fY116RE2EuySrE1zLyQ/0/index ... 5220 elasticsearch-6.5.1/data-0 ``` ---
**Author:** ![jacobot](https://avatars.discourse-cdn.com/v4/letter/j/4491bb/32.png) [@jacobot](https://discuss.elastic.co/u/jacobot)\ **Post date:** [December 4, 2018, 3:55pm UTC](https://discuss.elastic.co/t/storage-footprint-not-the-same-as-cummulative-index-size-why-is-that/158872/12 "2018-12-04T15:55:20Z")
Thank you @dadoonet and @DavidTurner Is there a way to look at the internals of the workings, would that be the `unallocated segments` you speak off? @DavidTurner, in your example of file listings, is the part of the path `Pw0fY116RE2EuySrE1zLyQ` the uuid of your index? ---
**Author:** ![jacobot](https://avatars.discourse-cdn.com/v4/letter/j/4491bb/32.png) [@jacobot](https://discuss.elastic.co/u/jacobot)\ **Post date:** [December 11, 2018, 4:18pm UTC](https://discuss.elastic.co/t/storage-footprint-not-the-same-as-cummulative-index-size-why-is-that/158872/13 "2018-12-11T16:18:47Z")
The problem is ocurring again with only 10gb in index data (total size disk is 50G) it stopped inserting in indexes, (it hit the 85% threshold) how can i correct this situation? i cannot imagine that 10gb disk space (single node), will eventually need a 50gb disk (mounted on /usr/share/elasticsearch/data) thanks you ---
**Author:** ![jacobot](https://avatars.discourse-cdn.com/v4/letter/j/4491bb/32.png) [@jacobot](https://discuss.elastic.co/u/jacobot)\ **Post date:** [December 11, 2018, 5:43pm UTC](https://discuss.elastic.co/t/storage-footprint-not-the-same-as-cummulative-index-size-why-is-that/158872/14 "2018-12-11T17:43:46Z")
feedback.. I tried this ```auto PUT /_all/_settings { "index":{ "translog.retention.age":"30m", "translog.retention.size":"100mb" } } ``` and this was the final result ```auto disk.indices disk.avail disk.used disk.total disk.percent 10.7gb 37.3gb 12.6gb 49.9gb 25 ``` All the numbers seem to add up now) `disk.indices` i snow very close to `disk.used` wich is excellent. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 8, 2019, 5:44pm UTC](https://discuss.elastic.co/t/storage-footprint-not-the-same-as-cummulative-index-size-why-is-that/158872/15 "2019-01-08T17:44:01Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.