@DavidTurner thanks for the quick response. We are seeing more performance changes in the cluster which seems to be related to mapping/bulk indexing changes.
We are using daily indexes with dynamic mappings so at the start of each day we have a lot of put mapping. In 6.2.4 this was never a big issue, for a few minutes the pending task will spike to up to 600 until most of the put mapping is over, In 7.4.2 we are seeing a much larger spike of 40,000 pending tasks that are causing the master to hang. The same data is being ingested to the cluster and the amount of nodes is equal.
The only thing that I can think of that is different, is that we copied indexes using a snapshot from version 6 to 7 and the old indexes had one mapping type.
It feels like the old version had some kind of limit that causes the nodes not to do all mapping in parallel and limit them to per thread request (by that throttling the pending task and protecting the master ).
Can you think of anything that changes in the bulk or put mapping area that could help us explain the new behavior we are seeing?
Thanks!