# Elasticsearch Transport Client bottle neck with concurrent calls

**URL:** <https://discuss.elastic.co/t/elasticsearch-transport-client-bottle-neck-with-concurrent-calls/65077>\
**Category:** Elasticsearch\
**Created:** [November 5, 2016, 1:53am UTC](https://discuss.elastic.co/t/elasticsearch-transport-client-bottle-neck-with-concurrent-calls/65077 "2016-11-05T01:53:20Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![shivangshah](https://avatars.discourse-cdn.com/v4/letter/s/ba9def/32.png) [@shivangshah](https://discuss.elastic.co/u/shivangshah)\
**Post date:** [November 5, 2016, 1:53am UTC](https://discuss.elastic.co/t/elasticsearch-transport-client-bottle-neck-with-concurrent-calls/65077/1 "2016-11-05T01:53:20Z")

</div>

We ran a performance test on my local environment with one node, one index and one shard. The number of TPS was around 60. We started low around 20 to 25 TPS and the 99th percentile was around 100ms which is what he expected. The moment we started getting close to 50 TPS, our requests started queuing, timing out (at http level) and failing. When we looked at the report, 50th percentile was around 20 seconds.

Created a github issue for this: [https://github.com/elastic/elasticsearch/issues/21349#issuecomment-258581935](https://github.com/elastic/elasticsearch/issues/21349#issuecomment-258581935) and was routed here

The problem is not the client threads being parked .. the problem is transport client being the bottle neck of not being able to handle many concurrent requests. VisualVM profiler tells us that almost all the time, the threads are waiting on the `BaseSync.get()` call.

Here's the gatling report.

 ![image](https://cloud.githubusercontent.com/assets/2624839/20026787/ad42aff8-a2be-11e6-916b-50b46f8f3dbc.png)  
The calls are going to one index and one shard (no replica). Everything is local as well.  
As you can see that the initial calls returned back almost instantaneously, but over time requests started getting queued up and eventually when you monitor through a profiler, you can see all threads are waiting (blocked?) on `Sync.get()`. I hope this sheds any light. And if anyone can point me to any performance tests that were done using transport client, that'd be great as well.

Anyone else saw this issue or did any further performance tests? I would expect the transport client to be async in all possible ways but seems like it blocking on calls.

All help appreciated!

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [November 5, 2016, 7:06am UTC](https://discuss.elastic.co/t/elasticsearch-transport-client-bottle-neck-with-concurrent-calls/65077/2 "2016-11-05T07:06:34Z")

</div>

What does your code look like?  
Are you calling get() or actionGet()?

Instead of execute()?

Calling get() do wait for the response. You can use a listener instead.

---

<div class="post-metadata">

**Author:** ![shivangshah](https://avatars.discourse-cdn.com/v4/letter/s/ba9def/32.png) [@shivangshah](https://discuss.elastic.co/u/shivangshah)\
**Post date:** [November 5, 2016, 7:49am UTC](https://discuss.elastic.co/t/elasticsearch-transport-client-bottle-neck-with-concurrent-calls/65077/3 "2016-11-05T07:49:33Z")

</div>

Actually I am using Observables. Here's the code sample I use (it's very close to the actual code, but not actual code)

```auto
private void multiSearch(MultiSearchRequestBuilder builder) {
        logger.debug("Executing multisearch query");
        long start = System.currentTimeMillis();
        Observable.defer(() -> Observable.from(builder.execute()))
            .map(multiSearchResponse -> {
                long end = System.currentTimeMillis();
                System.out.println("Total time took for query: " + Thread.currentThread().getName() + " " + (end - start));
                List<SearchResult> searchResults = new ArrayList<>();

                for (MultiSearchResponse.Item item : multiSearchResponse.getResponses()) {
                    SearchResult result = getSearchResultsFromSearchResponse(item.getResponse());
                    if (result.getHits().size() > 0) {
                        searchResults.add(result);
                    }
                }
    ...
}).subscribeOn(Schedulers.io())
.subscribe(new Subscriber<List<SearchResult>>() {
                @Override
                public void onCompleted() {
                    // when the stream is completed
                }

                @Override
                public void onError(Throwable throwable) {                    
                    deferredResult.setErrorResult(throwable);
                }

                @Override
                public void onNext(List<SearchResult> searchResults) {
                    try {
                        deferredResult.setResult(ResponseEntity.ok(objectMapper.writeValueAsString(searchResults)));
                    } catch (Exception e) {
                        throw new RuntimeException(e);
                    }
                }
            });

```

I ran the test again, this time for an hour. You can see that 50th percentile is great and than suddenly 75th percentile shoots up. Somewhere, when the load starts building, the transport client blocks. Because everythign else in the code is pretty much reactive based on the listenableFuture (which is converted to observable)

 ![](https://us1.discourse-cdn.com/elastic/original/2X/e/e37b5154dd83c7657328ef8042b18743f4882b91.png)

---

<div class="post-metadata">

**Author:** ![shivangshah](https://avatars.discourse-cdn.com/v4/letter/s/ba9def/32.png) [@shivangshah](https://discuss.elastic.co/u/shivangshah)\
**Post date:** [November 6, 2016, 8:46pm UTC](https://discuss.elastic.co/t/elasticsearch-transport-client-bottle-neck-with-concurrent-calls/65077/4 "2016-11-06T20:46:02Z")

</div>

So I tested and tested over the weekend and turns out the problem is not the client but it's the highlighting ! Let me start a different discussion on it and get more inputs.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:06pm UTC](https://discuss.elastic.co/t/elasticsearch-transport-client-bottle-neck-with-concurrent-calls/65077/5 "2017-07-05T22:06:19Z")

</div>