# Disk.indices is higher than disk.used in cat allocation **URL:** \ **Category:** Elasticsearch\ **Created:** [July 28, 2021, 11:54am UTC](https://discuss.elastic.co/t/disk-indices-is-higher-than-disk-used-in-cat-allocation/279853 "2021-07-28T11:54:19Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [July 28, 2021, 11:54am UTC](https://discuss.elastic.co/t/disk-indices-is-higher-than-disk-used-in-cat-allocation/279853/1 "2021-07-28T11:54:19Z")
When I run the following command: ```auto GET _cat/allocation?v&s=disk.indices&h=shards,disk.indices,disk.used,disk.available,disk.total,disk.percent ``` it shows the following output: ```auto shards disk.indices disk.used disk.total disk.percent 160 1.4tb 1.4tb 1.7tb 86 160 1.4tb 1.4tb 1.7tb 87 160 1.5tb 1.5tb 1.7tb 89 160 1.5tb 1.5tb 1.7tb 90 480 7.7tb 3.7tb 20tb 18 480 7.7tb 3.9tb 20tb 19 ``` Can anyone help me understand how come the `disk.indices` in last two rows **exceeds** `disk.used`? ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [July 29, 2021, 8:12am UTC](https://discuss.elastic.co/t/disk-indices-is-higher-than-disk-used-in-cat-allocation/279853/2 "2021-07-29T08:12:46Z")
Are you using searchable snapshots on those nodes? Just wondering if there is a difference, as they also hold more shards than the others. ---
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [July 29, 2021, 9:26am UTC](https://discuss.elastic.co/t/disk-indices-is-higher-than-disk-used-in-cat-allocation/279853/3 "2021-07-29T09:26:31Z")
Hey Alexander, No. Am not making use of searchable snapshots. ELK version is 7.10.2. It's pretty puzzling as to how `disk.indices` can be greater than `disk.used`. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [July 29, 2021, 2:39pm UTC](https://discuss.elastic.co/t/disk-indices-is-higher-than-disk-used-in-cat-allocation/279853/4 "2021-07-29T14:39:36Z")
This indeed looks like a bug then. Can you open an issue in [Issues · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues) please? ---
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [July 29, 2021, 3:52pm UTC](https://discuss.elastic.co/t/disk-indices-is-higher-than-disk-used-in-cat-allocation/279853/5 "2021-07-29T15:52:19Z")
Sure. Will do. The [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/cat-allocation.html#cat-allocation-api-response-body) states that > This metric double-counts disk space for hard-linked files, such as those created when [shrinking](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/indices-shrink-index.html), [splitting](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/indices-split-index.html), or [cloning](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/indices-clone-index.html) an index. I can the API `GET _tasks?detailed=true` and don't see any re-index or any other tasks. The only tasks that are listed seem to be of `cluster:monitor/tasks/`. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 29, 2021, 4:29pm UTC](https://discuss.elastic.co/t/disk-indices-is-higher-than-disk-used-in-cat-allocation/279853/6 "2021-07-29T16:29:56Z")
`disk.indices` is the total of the store sizes of each index (i.e. the sum of the sizes of the individual files), whereas `disk.used` is whatever the OS reports as the used space on the underlying filesystem. In particular that means that `disk.indices` will double-count any hard-linked files whereas `disk.used` won't. ---
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [July 29, 2021, 4:37pm UTC](https://discuss.elastic.co/t/disk-indices-is-higher-than-disk-used-in-cat-allocation/279853/7 "2021-07-29T16:37:45Z")
Thanks @DavidTurner . Would you be able to shed more light on `hard-linked` files here? How would they get created? AFAIK, this is a managed cluster so unlikely that anyone could manually create hard-links. Is there a possibility that say `shrink` was triggered and then aborted mid-way leading to hard-linked files being present. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 29, 2021, 4:44pm UTC](https://discuss.elastic.co/t/disk-indices-is-higher-than-disk-used-in-cat-allocation/279853/8 "2021-07-29T16:44:51Z")
Yeah things like shrink or split or clone are the usual answer. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [August 2, 2021, 7:27am UTC](https://discuss.elastic.co/t/disk-indices-is-higher-than-disk-used-in-cat-allocation/279853/9 "2021-08-02T07:27:25Z")
@DavidTurner do you still consider this a bug from user perspective and these confusing numbers being shown, despite the logic of the explanation of that behaviour? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 2, 2021, 7:53am UTC](https://discuss.elastic.co/t/disk-indices-is-higher-than-disk-used-in-cat-allocation/279853/10 "2021-08-02T07:53:44Z")
Not really, these numbers measure different things and are both important as they are defined today. `disk.indices` is a good measure of the size of the overall dataset and is insensitive to things like hard-linking, whereas `disk.used` tells us how much real disk space we're actually using right now, accounting for hard-linked files, filesystem overhead (e.g. rounding up small files to a whole block), non-file space usage (e.g. directory entries), and other data also stored on the same filesystem. The [docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-allocation.html) do spell out the difference. ---
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [August 2, 2021, 8:16am UTC](https://discuss.elastic.co/t/disk-indices-is-higher-than-disk-used-in-cat-allocation/279853/11 "2021-08-02T08:16:46Z")
Hi David, Thanks for the clarification. Makes sense. But I've a few questions: > [@DavidTurner](#): > > `disk.indices` is a good measure of the size of the overall dataset and is insensitive to things like hard-linking Is that why it double counts the hard-linked files? I wouldn't be surprised if `disk.used > disk.indices` but in this case, the `disk.indices > disk.used` since last 6 consecutive days (and probably even more). If there were a re-index / cloning / shrinking task going on, it would be listed under `GET _tasks?detailed=true`. Correct? But the output of `GET _tasks?detailed=true` just shows all tasks are of `cluster:monitor/tasks/`. How do we find the root cause of `disk.indices > disk.used` in a managed ES Cluster where we cannot `ssh` ? Any thoughts? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 2, 2021, 10:34am UTC](https://discuss.elastic.co/t/disk-indices-is-higher-than-disk-used-in-cat-allocation/279853/12 "2021-08-02T10:34:37Z")
> [@sandeepkanabar](#): > > Is that why it double counts the hard-linked files? Yes. > [@sandeepkanabar](#): > > If there were a re-index / cloning / shrinking task going on, it would be listed under `GET _tasks?detailed=true` . Correct? Reindex yes, but that's not relevant. Cloning and shrinking are pretty quick, but will keep using hard-linked files for arbitrarily long, so you probably wouldn't see anything about them in the tasks. > [@sandeepkanabar](#): > > How do we find the root cause Maybe this is the fundamental point: is there actually a problem here? Do you need to investigate? ---
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [August 3, 2021, 10:00am UTC](https://discuss.elastic.co/t/disk-indices-is-higher-than-disk-used-in-cat-allocation/279853/13 "2021-08-03T10:00:51Z")
Thanks David. I thought it was a problem until you clarified in a previous post. I was looking into the huge disk space occupied by this cluster and while `disk.used` showed `30TB`, `disk.indices` reported `37.2TB`. So I was puzzled which one to rely on since `7.2TB` difference is a huge number. > [@DavidTurner](#): > > `disk.used` tells us how much real disk space we're actually using right now, accounting for hard-linked files, filesystem overhead (e.g. rounding up small files to a whole block), non-file space usage (e.g. directory entries), and other data also stored on the same filesystem So I suppose I can rest assure that this cluster uses `30TB` and not `37.2TB`. Correct? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 3, 2021, 11:02am UTC](https://discuss.elastic.co/t/disk-indices-is-higher-than-disk-used-in-cat-allocation/279853/14 "2021-08-03T11:02:35Z")
Correct, `disk.used` is the actual current disk usage. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 31, 2021, 11:03am UTC](https://discuss.elastic.co/t/disk-indices-is-higher-than-disk-used-in-cat-allocation/279853/15 "2021-08-31T11:03:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.