Unbalanced disk usage - ES 2.1


(Christian Dahlqvist) #21

I do not know. Have not used Elasticsearch 2.x in a long time...


(Wédney Yuri) #22

In your case seems to be related to https://github.com/elastic/elasticsearch/pull/29125 which was fixed in version 6.2.4.

We also noticed a huge increase in open file descriptors:

image


(Yoni Farin) #23

Hi, @Wedney_Yuri I think the issue you described is related to https://github.com/elastic/elasticsearch/issues/29488.
We are experiencing a similar issue with 6.2.4 as described in https://github.com/elastic/elasticsearch/issues/30351

Hopefully, the following merge that will be release in 6.3.0 will solve the issue:


(Wédney Yuri) #24

Thank you, I will investigate.


(Wédney Yuri) #25

You're right, we're noticing full disk usage in version 6.2.4. How did you solve this problem?


(Yoni Farin) #26

As far as I know there is no solution and we are left to wait for 6.3.0 to be released.
Calling force merge didn't help. The only two things that cause the shards to shrink were:

  1. Closing and reopen the index
  2. Restart the node that the shards is on

These actions will force elastic to initialize the shards and only then it will run a successful merge on these shards.
Both are not a valid workaround :frowning:

Note that the initialize of these shards can take ages depends on the shard size, during this time the data in these shards won't be accessible.


(David Turner) #27

It should be sufficient to set the number of replicas to zero and back to 1, which will cause the replicas to be reinitialised. It is also even possible to use the reroute API to cancel the allocation of the problematic replica which will cause just that one shard to be rebuilt.


(Yoni Farin) #28

In our case, the shards that didn't perform merge operations were the primaries and removing the replicas didn't do the trick.


(Wédney Yuri) #29

As @farin99 said, it's happening to primary nodes as well.

This has been happening almost every day, do you know which is the last stable version of elasticsearch that was released without this bug?


(Wédney Yuri) #30

@farin99 I don't know why, but we changed indices.recovery.max_bytes_per_sec to 4000mb and the disk usage was reduced after minutes.


(system) #31

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.