Cluster routing allocation decider preventing allocation

Elastic Stack Elasticsearch
1

I have 3 ElasticSearch servers in the cluster. Out of three, two are working normally and one server behaving strange. One server have no shard on it. When i restart the elasticsearch service or reboot the server, it is giving me the below error:

[WARN ][cluster.routing.allocation.decider] [testing_node] after allocating, node [testing_node] would have less than the required 0b free bytes threshold (-22254411079 bytes free), preventing allocation

Although i have almost 98% free disk space on the server

2

Hmm, weird.

First up, there's a display bug that was fixed in #33641 (v6.5.0). The number reported (negative 22GB) is coming from the calculated disk space minus the estimated size of the shard. In ≥ 6.5.0 we show these two numbers separately which is a bit more useful. From this message it's not clear if it's estimating the size of the disk as too small or the size of the shard as too large.

Still, this seems unexpected. If you do this...

POST _cluster/settings
{"persistent":{"logger.org.elasticsearch.cluster.routing.allocation.decider":"TRACE"}}

... then you should get copious logs from the allocator that might shed more light on what's going on. Reset this to normal with this...

POST _cluster/settings
{"persistent":{"logger.org.elasticsearch.cluster.routing.allocation.decider":null}}
3

Thank you David, for your guidance. It started working now.

I ran the above command and checked the log file. Nothing was logged. Then I checked the _cluster/health?pretty and it was showing me "relocating_shards" : 2.

I'm pretty sure that only running the above command haven't fixed my issue.

4

Hi David,

I'm not able to reset it. when i run the command:

POST _cluster/settings
{"persistent":{"logger.org.elasticsearch.cluster.routing.allocation.decider":null}}

it is giving me below error:

{"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: no settings to update;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: no settings to update;"},"status":400}

5

Apologies, I mean PUT rather than POST. What version are you using?

6

Tried PUT as well, no success

Request
curl -XPUT localhost:9200/_cluster/settings -d '{"persistent":{"logger.org.elasticsearch.cluster.routing.allocation.decider":null}}'

Response:
{"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: no settings to update;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: no settings to update;"},"status":400}

ES version is 2.3.3

7

Ohhhh. That's very old. It went end-of-life in September 2017. I don't have access to a development environment in which I can dig into the problem you're facing, but it could well be addressed in a more recent version. I recommend upgrading.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.