Why does Elasticsearch choose to relocate shards?

Elastic Stack Elasticsearch
1

I've read https://www.elastic.co/blog/every-shard-deserves-a-home but can't see how any of the cases there apply.

ES seems to be spending all its time moving shards around, sometimes apparently taking hours to move multi-gigabyte shards, occupying both the two-things-at-once that it seems to like doing, thus other, smaller, shards remain "yellow" for hours waiting for a chance to run.

How do I find out what's going on and why?

2

What version are you on?

3

6.2.4

4

It seems very odd that it would be continually moving things.

How much disk space do you have on your nodes?

5
  1. 165G, 81% used
  2. 165G, 71% used
  3. 165G, 53% used
6

It might be moving them from the first node due to watermarks - https://www.elastic.co/guide/en/elasticsearch/reference/6.4/disk-allocator.html

What do the logs show?

7

I assumed it wouldn't be that because the disk usage is below 85%. But ... the largest shards it moves around are 7.9G, which is more than 4% of 165G, and 81% + 4.7% > 85% so perhaps that is indeed what's going on, although I've never seen actual usage above 85%.

Thank you for pointing me at the logs - I deleted all the historical logs from the node that was at 81% and it's now at 71% ... the logs themselves are full of errors which I supposed I'd better clean up before trying to find anything in there about relocating shards.

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.