We're having difficulty with RestClient against a large 700 node ES cluster. Under a moderate search load (47 concurrent searches) we see
- extensive use of I/O Dispatchers (based on the default I/O Recator configs in the httpclient lib you use
- apparent queuing of requests
We don't understand the DEFAULT_MAX_CONN_TOTAL vs. DEFAULT_MAX_CONN_PER_ROUTE settings you default to in RestClientBuilder.java. I've been scouring httpclient/core source code to understand.
Main question: when we create a new RestClient against a SLOW ES cluster (can take hours to retrieve 250k docs) - as we call performRequest() over scroll ids - are we creating new connections?
We can't recreate the problem at our office under smaller clusters - unless we can "slow down" ES per query - is there some secret thing we can do to make ES take minutes per query/page/scroll vs. ms/seconds just to keep these connections open longer to test the JVM/threading issues?