Compress elasticsearch bulkrequest using java high level rest client and bulk processor

Elastic Stack Elasticsearch
1

Is there anyway to compress the elasticsearch bulk request payload using java high level rest client and bulk processor?

I have tried below

requestConfigBuilder.setContentCompressionEnabled(true);

But it does not seem to be compressing the request payload.
Do I need to enable any other settings?

Any inputs?

2

Hey, any chance to create a code snippet that can be tested in an IDE and follow the steps from your setup?

Thank you!

3 
 RestClientBuilder restClientBuilder = RestClient.builder(httpHosts);

        SniffOnFailureListener sniffOnFailureListener = new SniffOnFailureListener();
        if (!config.isEsNodeSniffingDisabled()) {
            restClientBuilder.setFailureListener(sniffOnFailureListener);
        }

        restClientBuilder.setHttpClientConfigCallback(httpClientBuilder -> {
            httpClientBuilder
                    .setDefaultConnectionConfig(ConnectionConfig.custom().setCharset(StandardCharsets.UTF_8).build());
            return httpClientBuilder;
        });

        restClientBuilder.setRequestConfigCallback(requestConfigBuilder -> {
            requestConfigBuilder.setSocketTimeout(config.getEsSocketTimeout()*1000);
            requestConfigBuilder.setContentCompressionEnabled(true);
            requestConfigBuilder.setConnectTimeout(config.getEsConnectionTimeout()*1000);
            requestConfigBuilder.setConnectionRequestTimeout(config.getEsConnectionTimeout()*1000);
            return requestConfigBuilder;
        });
 return new RestHighLevelClient(restClientBuilder);
4

Just to give more context, We were trying to do load test on es cluster (with 150 data nodes) deployed on AWS EC2 instances. We were not able to get the desired ingestion throughput despite making all the recommended settings for better indexing performance.
Then we saw that the problem was with the network bandwidth. The AWS i3.2x large instances come with Up to 10 Gigabit/second network bandwidth and this network was clogged.
So we were not able to ingest the data at a better throughput. So, I was thinking we could improve the throughput by compressing the bulk payload. I could not find any documentation on how to compress the payload using java high level rest client . Any inputs would be really helpful.

5

I believe that the only effect of this line is to indicate to Elasticsearch that the client will accept a compressed response. If you want the client to compress the requests too you should call RestClientBuilder#setCompressionEnabled instead.

Note that compression takes substantial extra CPU effort. An i3.2xlarge only has 8 CPUs, I think you'd need several times that number to reach 10Gbps of throughput.

6

Thank you @DavidTurner ,
Let me try this and see if I see any improvement in throughput. Thank you!

7

@DavidTurner ,
I see that this method

RestClientBuilder#setCompressionEnabled

is available only from elasticsearch-rest-client version 7.10.1.
But we are running elasticsearch server version 7.8.0.
Is it possible to use rest client version 7.10.1 to compress bulk request and ingest into elasticsearch server 7.8.0?

8

Ah yes, it was introduced in 7.10.0 (see below). I don't know another way to do it with the high-level REST client, although of course it's pretty simple to use a bare HTTP client to send bulk requests with compression.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.