Elasticsearch don't remove old shards

Elastic Stack Elasticsearch
1

Hi. This week after _forcemerge?only_expunge_deletes=true I saw an increase in disk space usage (# 1 in the picture), after that I tried to reduce the number of replicas from 2 to 1 (# 2 in the picture), and also noticed an increase in disk space usage. After returning the number of replicas, the disk space usage also increases (# 3 in the picture).

Screenshot 2021-09-15 at 14.30.58
Screenshot 2021-09-15 at 14.30.582478×960 205 KB

After removed some count of replicas i expected reduce disk usage. After return count of replicas back i expected the same disk usage which was before, but usage only increase and increase.

via elasticsearch api i see that available only two shards but in data directory i see four.

Can anyone help me understand why this is occur and how to avoid it in future?

es version: 6.8.13
number of nodes: 30
number of pr shards: 20
number of replicas: 2

2

Welcome to our community! :smiley:

How large, in GB, is the index?

3

Thx. The total size of shards (with replicas) is about 100 GB.

4

hi :raised_hand: after a week the disk space has not changed. any ideas?

5

I remember some old bugs where the translog was not cleaned as expected.
May be upgrading to 6.8.19 would help?

Did you try also to forcemerge the segment count to 1?

6

Thx, for ur answer :slight_smile: I have checked that vastest majority of disk space usage spend shard directories which are not present in elasticsearch.

8.0K	./_state
6.2G	./4
4.1G	./16
3.9G	./17
6.2G	./5

elasticsearch api shows that only shards numbered 16 and 17 exists on the current host, but on fs I have 2 and 5. After check directory 2 and 5 i have found full lucene's index with liv, cfs, cfe files (translog spend not a lot of space).

Did you try also to forcemerge the segment count to 1?

No, i didn't (it's bad for inserting in my case), but i did _forcemerge?only_expunge_deletes=true which only increase disk usage on host.

I'm really not understand why elasticsearch not remove old relocated shards. Does elasticsearch api has some endpoint for explicit triggering removing unnecessary data?

7

That looks weird.

May be Cluster reroute API | Elasticsearch Guide [8.11] | Elastic could help but I don't have enough experience on that.
@DavidTurner Do you know?

I guess it's unsafe to remove manually the shards which should not be there according to the shard allocation table...

8

Elasticsearch automatically cleans up unused shard data when the shard health is green and none of its copies are relocating. If it can't (e.g. something forbids it from deleting the data) then it will log a warning about the failure.

9

@dadoonet thx for suggestion.

The reroute command allows for manual changes to the allocation of individual shards in the cluster.

if i understand correct then current api help to do something with available shards. In my case shard is not available.

@DavidTurner i have found warning related with my shards

[2021-09-14T17:20:00,578][INFO ][o.e.i.s.TransportNodesListShardStoreMetaData] [elasticsearch-node8] [index_v12][13]: failed to obtain shard lock
org.elasticsearch.env.ShardLockObtainFailedException: [index_v12][13]: obtaining shard lock timed out after 5000ms

and that's all

I have checked _cluster/allocation/explain?pretty route but all is fine (no one unassigned shard). Any ideas?

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.