I managed to reproduce a scenario where using Transport Client 6.2.4 to send about 100
heavy search requests (heavy aggregation searches that hit many shards on the targeted cluster)
with action listeners (every search call is an async call with very simple ActionListener that updates an atomic counter on both callbacks - success and failure - and returns immediately), results in not all action listeners actually invoked.
It's important to note that this happens only in a cross-cluster search setup. In our setup, we have a small cluster, that is actually a proxy to 4 other big clusters. All search requests are going to that proxy cluster, but depending on the indices in the search request, the search will be delegated to one of the four other clusters.