# Elasticsearch don't remove old shards **URL:** \ **Category:** Elasticsearch\ **Created:** [September 15, 2021, 11:05am UTC](https://discuss.elastic.co/t/elasticsearch-dont-remove-old-shards/284272 "2021-09-15T11:05:58Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![09a9373743777d8430c4](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/09a9373743777d8430c4/32/94606_2.png) [@09a9373743777d8430c4](https://discuss.elastic.co/u/09a9373743777d8430c4)\ **Post date:** [September 15, 2021, 11:05am UTC](https://discuss.elastic.co/t/elasticsearch-dont-remove-old-shards/284272/1 "2021-09-15T11:05:59Z")
Hi. This week after \_forcemerge?only\_expunge\_deletes=true I saw an increase in disk space usage (# 1 in the picture), after that I tried to reduce the number of replicas from 2 to 1 (# 2 in the picture), and also noticed an increase in disk space usage. After returning the number of replicas, the disk space usage also increases (# 3 in the picture). ![Screenshot 2021-09-15 at 14.30.58](https://us1.discourse-cdn.com/elastic/original/3X/f/d/fd7108bd27b14c29c01e7e79fcda4e14d544e57e.png) After removed some count of replicas i expected reduce disk usage. After return count of replicas back i expected the same disk usage which was before, but usage only increase and increase. via elasticsearch api i see that available only two shards but in data directory i see four. Can anyone help me understand why this is occur and how to avoid it in future? es version: 6.8.13 number of nodes: 30 number of pr shards: 20 number of replicas: 2 ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 15, 2021, 10:05pm UTC](https://discuss.elastic.co/t/elasticsearch-dont-remove-old-shards/284272/2 "2021-09-15T22:05:24Z")
Welcome to our community! 😃 How large, in GB, is the index? ---
**Author:** ![09a9373743777d8430c4](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/09a9373743777d8430c4/32/94606_2.png) [@09a9373743777d8430c4](https://discuss.elastic.co/u/09a9373743777d8430c4)\ **Post date:** [September 16, 2021, 9:39am UTC](https://discuss.elastic.co/t/elasticsearch-dont-remove-old-shards/284272/3 "2021-09-16T09:39:53Z")
Thx. The total size of shards (with replicas) is about 100 GB. ---
**Author:** ![09a9373743777d8430c4](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/09a9373743777d8430c4/32/94606_2.png) [@09a9373743777d8430c4](https://discuss.elastic.co/u/09a9373743777d8430c4)\ **Post date:** [September 23, 2021, 2:56pm UTC](https://discuss.elastic.co/t/elasticsearch-dont-remove-old-shards/284272/4 "2021-09-23T14:56:52Z")
hi ✋ after a week the disk space has not changed. any ideas? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 23, 2021, 3:29pm UTC](https://discuss.elastic.co/t/elasticsearch-dont-remove-old-shards/284272/5 "2021-09-23T15:29:06Z")
I remember some old bugs where the translog was not cleaned as expected. May be upgrading to 6.8.19 would help? Did you try also to forcemerge the segment count to 1? ---
**Author:** ![09a9373743777d8430c4](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/09a9373743777d8430c4/32/94606_2.png) [@09a9373743777d8430c4](https://discuss.elastic.co/u/09a9373743777d8430c4)\ **Post date:** [September 23, 2021, 6:36pm UTC](https://discuss.elastic.co/t/elasticsearch-dont-remove-old-shards/284272/6 "2021-09-23T18:36:05Z")
Thx, for ur answer 🙂 I have checked that vastest majority of disk space usage spend shard directories which are not present in elasticsearch. ```auto 8.0K ./_state 6.2G ./4 4.1G ./16 3.9G ./17 6.2G ./5 ``` elasticsearch api shows that only shards numbered 16 and 17 exists on the current host, but on fs I have 2 and 5. After check directory 2 and 5 i have found full lucene's index with liv, cfs, cfe files (translog spend not a lot of space). > Did you try also to forcemerge the segment count to 1? No, i didn't (it's bad for inserting in my case), but i did `_forcemerge?only_expunge_deletes=true` which only increase disk usage on host. I'm really not understand why elasticsearch not remove old relocated shards. Does elasticsearch api has some endpoint for explicit triggering removing unnecessary data? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 28, 2021, 11:53am UTC](https://discuss.elastic.co/t/elasticsearch-dont-remove-old-shards/284272/7 "2021-09-28T11:53:32Z")
That looks weird. > [@09a9373743777d8430c4](#): > > Does Elasticsearch api has some endpoint for explicit triggering removing unnecessary data? May be [Cluster reroute API | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html) could help but I don't have enough experience on that. @DavidTurner Do you know? I guess it's unsafe to remove manually the shards which should not be there according to the shard allocation table... ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [September 28, 2021, 12:28pm UTC](https://discuss.elastic.co/t/elasticsearch-dont-remove-old-shards/284272/8 "2021-09-28T12:28:50Z")
Elasticsearch automatically cleans up unused shard data when the shard health is green and none of its copies are relocating. If it can't (e.g. something forbids it from deleting the data) then it will log a warning about the failure. ---
**Author:** ![09a9373743777d8430c4](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/09a9373743777d8430c4/32/94606_2.png) [@09a9373743777d8430c4](https://discuss.elastic.co/u/09a9373743777d8430c4)\ **Post date:** [September 28, 2021, 3:20pm UTC](https://discuss.elastic.co/t/elasticsearch-dont-remove-old-shards/284272/9 "2021-09-28T15:20:56Z")
@dadoonet thx for suggestion. > The reroute command allows for manual changes to the allocation of individual shards in the cluster. if i understand correct then current api help to do something with available shards. In my case shard is not available. @DavidTurner i have found warning related with my shards ```auto [2021-09-14T17:20:00,578][INFO][o.e.i.s.TransportNodesListShardStoreMetaData] [elasticsearch-node8] [index_v12][13]: failed to obtain shard lock org.elasticsearch.env.ShardLockObtainFailedException: [index_v12][13]: obtaining shard lock timed out after 5000ms ``` and that's all I have checked `_cluster/allocation/explain?pretty` route but all is fine (no one unassigned shard). Any ideas? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 26, 2021, 3:21pm UTC](https://discuss.elastic.co/t/elasticsearch-dont-remove-old-shards/284272/10 "2021-10-26T15:21:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.