# Understanding RestClient.java and its use of DEFAULT\_MAX\_CONN\_TOTAL **URL:** \ **Category:** Elasticsearch\ **Tags:** language-clients\ **Created:** [December 14, 2020, 2:03pm UTC](https://discuss.elastic.co/t/understanding-restclient-java-and-its-use-of-default-max-conn-total/258583 "2020-12-14T14:03:51Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![bdbusches](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bdbusches/32/80815_2.png) [@bdbusches](https://discuss.elastic.co/u/bdbusches)\ **Post date:** [December 14, 2020, 2:03pm UTC](https://discuss.elastic.co/t/understanding-restclient-java-and-its-use-of-default-max-conn-total/258583/1 "2020-12-14T14:03:51Z")
We're having difficulty with RestClient against a large 700 node ES cluster. Under a moderate search load (47 concurrent searches) we see - extensive use of I/O Dispatchers (based on the default I/O Recator configs in the httpclient lib you use - apparent queuing of requests We don't understand the DEFAULT\_MAX\_CONN\_TOTAL vs. DEFAULT\_MAX\_CONN\_PER\_ROUTE settings you default to in RestClientBuilder.java. I've been scouring httpclient/core source code to understand. Main question: when we create a new RestClient against a SLOW ES cluster (can take hours to retrieve 250k docs) - as we call performRequest() over scroll ids - are we creating new connections? We can't recreate the problem at our office under smaller clusters - unless we can "slow down" ES per query - is there some secret thing we can do to make ES take minutes per query/page/scroll vs. ms/seconds just to keep these connections open longer to test the JVM/threading issues? Thanks ---
**Author:** ![bdbusches](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bdbusches/32/80815_2.png) [@bdbusches](https://discuss.elastic.co/u/bdbusches)\ **Post date:** [December 14, 2020, 4:14pm UTC](https://discuss.elastic.co/t/understanding-restclient-java-and-its-use-of-default-max-conn-total/258583/2 "2020-12-14T16:14:26Z")
I understand it a bit more - i enabled org.apache.http debug - I can see that we clearly didn't understand the httpconfig and the defaults used in your RestClient (max of 10 per route, 30 total). Plus, the interaction with the default IO Reactor in all this is horrible - we're at http 4x, so the default is to create NCORES \* 2 "I/O dispatcher" threads. For us, thats 160 per connection. With just 47 concurrent searches we have nearly 7500 threads created to do IO! Plus, with the max route/total limits, we're not even sure what's happening - how are they queuing, etc. Appreciate any light. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 14, 2020, 4:58pm UTC](https://discuss.elastic.co/t/understanding-restclient-java-and-its-use-of-default-max-conn-total/258583/3 "2020-12-14T16:58:21Z")
> [@bdbusches](#): > > as we call performRequest() over scroll ids - are we creating new connections? No, the REST client should be re-using connections. > [@bdbusches](#): > > For us, thats 160 per connection. With just 47 concurrent searches we have nearly 7500 threads created to do IO! That sounds very wrong. Are you creating a client per search? You should only normally have a single instance of the client in existence, and it should normally live until your application shuts down. ---
**Author:** ![bdbusches](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bdbusches/32/80815_2.png) [@bdbusches](https://discuss.elastic.co/u/bdbusches)\ **Post date:** [December 16, 2020, 11:31pm UTC](https://discuss.elastic.co/t/understanding-restclient-java-and-its-use-of-default-max-conn-total/258583/4 "2020-12-16T23:31:52Z")
> [@DavidTurner](#): > > That sounds very wrong. Are you creating a client per search? You should only normally have a single instance of the client in existence, and it should normally live until your application shuts down. it is wrong - our app is a GIS app which supports dynamic searches (user driven) of thousands of different layers - all from about 10 types of connections - ES, PostgreSQL, MemSQL, REST/WFS, MongoDB, etc.). So, our task manager handles queing and working off searches from a java thread pool. by default, it creates a NEW restClient for each search. Similar to how we create a new PostgreSQL/PostGIS connection per search (which is fine/expected for that kind of data source). Thanks, we're building into the code now a capability to keep a static restClient, and set all those configs. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 17, 2020, 6:07am UTC](https://discuss.elastic.co/t/understanding-restclient-java-and-its-use-of-default-max-conn-total/258583/5 "2020-12-17T06:07:13Z")
> [@bdbusches](#): > > Similar to how we create a new PostgreSQL/PostGIS connection per search (which is fine/expected for that kind of data source). I'm not sure that's good either, at least it wasn't when I last used PostgreSQL. Each connection to PostgreSQL spawns its own backend process which can itself take many milliseconds and definitely runs into problems with high numbers of concurrent clients. There are workarounds like pgBouncer, but IMO that's a bit of a hack compared with using an in-app connection pool. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 14, 2021, 6:07am UTC](https://discuss.elastic.co/t/understanding-restclient-java-and-its-use-of-default-max-conn-total/258583/6 "2021-01-14T06:07:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.