Pending Tasks increase more quickly and master do garbagge collector with failures

[mcortinas (Marc Cortinas Val)]

Hi,
I'm using ES 1.7.3 with 4 client nodes, 16 data nodes and 5 masters. We have detected a bug in our java cliente application this bug apply re-map for each index adding a mapping massivelly.

The cluster pending-task increase more quickly and the amount of the young heap space is used and finally the master do gabagge collection with failures.

The only way to stop it has been closing the indexes affected or dropping the http requests to these indexes ( blocking the URIs /index_pattern)

2 sugggestions:

• Maybe it will be useful some feature for close some index if the pending task for this index grow up more quickly.
• Another useful suggestion will be manage the amount of memory in masters to manage the pending tasks for each index

Obviouslly these are suggestions due a defect of our client, but I think it could be useful for system administrators.

Best,

Marc

[warkolm (Mark Walkom)]

You are welcome to raise these as feature requests on github, but I don't see how this would be useful for ES to be honest.

@mcortinas

Maybe it will be useful some feature for close some index if the pending task for this index grow up more quickly.

Then we'd have to track pending tasks, their rate of change, and what indices they apply to. Then would we need to open them again if it drops? This would all need to be written as it doesn't exist.

@mcortinas

Another useful suggestion will be manage the amount of memory in masters to manage the pending tasks for each index

What do you mean here? You want a management threadpool and that needs to be then managed somehow?

@mcortinas

Obviouslly these are suggestions due a defect of our client

The easier way is to use strict mappings - Dynamic Mapping | Elasticsearch: The Definitive Guide [2.x] | Elastic

Also;

@mcortinas

5 masters

There's no difference to the cluster when you have 5 masters compared to 3, so that's a bit of a waste.