I'm running into some weird issues with indexing a lot of documents using the java bulk processor in async mode. It runs well but we've noticed it deadlocking. Essentially running a thread dump it appears its waiting for another thread to complete in order for the current thread to get access to the semaphore. From what I can observe the cluster never responds back and is forever in a locked stage.
In terms of our cluster it has a high number of aliases and mappings per index, which we are looking now to reduce and refactor. But in the meantime, I can't understand why we would observe deadlocks where the cluster never responds back with a timeout. The ES 2.3 cluster seems to be much more prone to this issue, since doesn't have master only/data only setup. The 5.X cluster seems ATM ok with a restart. Nothing in the logs jump out that an issue has occurred.
I guess I'm trying to pinpoint the root cause, such that I can determine it is indeed in fact we have too many aliases and mappings.