# Rest High level client performance

**URL:** <https://discuss.elastic.co/t/rest-high-level-client-performance/159735>\
**Category:** Elasticsearch\
**Created:** [December 6, 2018, 1:24pm UTC](https://discuss.elastic.co/t/rest-high-level-client-performance/159735 "2018-12-06T13:24:45Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![gzeskas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gzeskas/32/38511_2.png) [@gzeskas](https://discuss.elastic.co/u/gzeskas)\
**Post date:** [December 6, 2018, 1:24pm UTC](https://discuss.elastic.co/t/rest-high-level-client-performance/159735/1 "2018-12-06T13:24:45Z")

</div>

Hi

I'm doing some research how to utilize elasticsearch cluster in best way.  
I have written simple java application that is using `Rest High level client` and setup of 2 elasticsearch nodes with 1 index that contains 2 shards and 0 replicas.

The problems that I encountered is that performance is not increasing with second node as I had around 7000 index request per second with 1 node and 1 shard the same performance I receive with 2 nodes and 2 shards with 0 replicas.

The code that it's quite simple:

```java
Flux.range(1, 1000000000)
    .flatMap(this::createNewIndexRequest)
    .flatMap(this::index)
    .subscribe(next -> {
        completed.addAndGet(batchSize);
    }, error -> {
        logger.error("Received error: ", error);
    }, () -> {
        logger.info("Completed");
    });

```

Method that is performing send operation

```java
    private Mono<IndexResponse> index(IndexRequest indexRequest) {
        return Mono.create(monoSink -> {
            client.indexAsync(indexRequest, RequestOptions.DEFAULT, new ActionListener<IndexResponse>() {
                @Override
                public void onResponse(IndexResponse indexResponse) {
                    monoSink.success(indexResponse);
                }

                @Override
                public void onFailure(Exception e) {
                    logger.error("Encounter problem during document indexing:", e);
                    monoSink.error(e);
                }
            });
        });
    }

```

Rest High level client is builded like this:

```java
    public RestClientBuilder restClientBuilder() {
        HttpHost[] hosts = this.properties.getUris().stream().map(HttpHost::create)
                .toArray(HttpHost[]::new);

        RestClientBuilder builder = RestClient.builder(hosts);
        builder.setHttpClientConfigCallback(b -> b.setDefaultHeaders(
                Collections.singleton(new BasicHeader(HttpHeaders.ACCEPT_ENCODING, "gzip"))))
                .setRequestConfigCallback(b -> b.setContentCompressionEnabled(true));

        builder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
            @Override
            public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
                httpClientBuilder.setMaxConnPerRoute(1000);
                httpClientBuilder.setMaxConnTotal(10000);
                return httpClientBuilder;
            }
        });
        return builder;
    }

```

Client is configured to use those 2 nodes addresses.

What I'm missing ? Maybe someone could point me where I should look for the information ?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 6, 2018, 1:57pm UTC](https://discuss.elastic.co/t/rest-high-level-client-performance/159735/2 "2018-12-06T13:57:39Z")

</div>

First thing you should do is to use the BULK API: [https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-document-bulk.html](https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-document-bulk.html)

Specifically in Java I'd recommend using the Bulk Processor: [https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-document-bulk.html#java-rest-high-document-bulk-processor](https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-document-bulk.html#java-rest-high-document-bulk-processor)

---

<div class="post-metadata">

**Author:** ![gzeskas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gzeskas/32/38511_2.png) [@gzeskas](https://discuss.elastic.co/u/gzeskas)\
**Post date:** [December 6, 2018, 3:51pm UTC](https://discuss.elastic.co/t/rest-high-level-client-performance/159735/3 "2018-12-06T15:51:24Z")

</div>

In my case it's impossible to use Bulk API, I have a case where it's needed to process message by message and index them.

Me previous numbers were incorrect as I was bound to CPU on local machine, when I tried the same on cloud machine with more CPU i received results that 1 node could handle around 16k index requests but after adding second node and second shard throughput only jumped to around 21.5k.

So i'm trying to figure out why I'm not able to receive lineare scalability ?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 6, 2018, 6:32pm UTC](https://discuss.elastic.co/t/rest-high-level-client-performance/159735/4 "2018-12-06T18:32:32Z")

</div>

> [@gzeskas](#):
>
> In my case it's impossible to use Bulk API, I have a case where it's needed to process message by message and index them.

Why that? What is the difference an `indexAsync` operation?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 3, 2019, 6:32pm UTC](https://discuss.elastic.co/t/rest-high-level-client-performance/159735/5 "2019-01-03T18:32:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.