Cluster Yellow - Heap size overload

Elastic Stack Elasticsearch
1

Hi,

An ILM policy had a conflict with curator last night. It wasn't working because of closed index.
I reopen it and retry ilm policy and seems ok.

But my cluster still yellow and cannot reallocate shard because of this :

Data too large, data for [<transport_request>] would be [30660165828/28.5gb], which is larger than the limit of [30601641984/28.5gb], real usage: [30660162168/28.5gb], new bytes reserved: [3660/3.5kb], usages [request=0/0b, fielddata=7205127741/6.7gb, in_flight_requests=3660/3.5kb, accounting=10134822598/9.4gb]]; ], allocation_status[no_attempt]]]"

I think it is because of too hight heap size memory.

How can I reduce heap size memory ? Close indexes can reduce it ?

Best regards,

Thomas R.

2

I found the problem.

I mixed curator and ILM for index management.

An ILM policy has crashed because of a closed index and grown up until 673Gb.

It started to saturate memory JVM. I reopened closed index and retry ILM policy. This solved ILM problem and a new index was created.

But this was not enough. The cache had to be clean to allow reassigning. After a cleaning and a rerouting process, the cluster is now green.

Here are the command which help me to solve it :slight_smile: :

Get allocation explanation

GET /index-000170/_ilm/explain

Open closed index

POST /index-000143/_open

Retry ilm policy :

POST /index-000170/_ilm/retry

Get stat of fieldata :

GET _stats/fielddata?fielddata_fields=*

Clear cache fielddata :

POST /index-00016*/_cache/clear

Restart failed allocation

POST _cluster/reroute?retry_failed=true

Sources :

I hope this can help someone a day. :slight_smile:

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.