Slow update-mapping pending tasks

Elastic Stack Elasticsearch
1

We have a multi-tenant index with a large number of mapped fields - the mapping JSON weighs in at ~47mb.

When fields are dynamically added to the mapping, the update-mapping pending task takes 1 minute or more to complete:

Screen Shot 2016-01-12 at 10.32.39.png680×558 80.1 KB

When multiple fields are dynamically mapped this can cause many queued update-mapping tasks as you can see in the image. The long-running tasks block other tasks and seem to be impairing performance of the cluster.

I'm aware that 2.x introduces mapping deltas which should make propagating changes more efficient. We are not yet on 2.x though. Is there anything I can do to remedy this on 1.7, short of cutting down on the number of mapped fields? Would upgrading to 2.x fix this performance bottleneck?

Also, why are the update-mapping tasks taking a minute or more to complete? I understand 47mb is quite large for the mapping definition but seems like it should be faster than that.

2

That's pretty big....

No and yes.

Every single change needs to send the entire changed cluster state over to each node and have it acknowledged. Cluster state is more than just the mapping, so it'll be more than 47MB.