# Compress elasticsearch bulkrequest using java high level rest client and bulk processor **URL:** \ **Category:** Elasticsearch\ **Tags:** language-clients\ **Created:** [August 4, 2021, 12:12pm UTC](https://discuss.elastic.co/t/compress-elasticsearch-bulkrequest-using-java-high-level-rest-client-and-bulk-processor/280425 "2021-08-04T12:12:29Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Praveen\_Atluri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/praveen_atluri/32/39721_2.png) [@Praveen\_Atluri](https://discuss.elastic.co/u/Praveen_Atluri)\ **Post date:** [August 4, 2021, 12:12pm UTC](https://discuss.elastic.co/t/compress-elasticsearch-bulkrequest-using-java-high-level-rest-client-and-bulk-processor/280425/1 "2021-08-04T12:12:29Z")
Is there anyway to compress the elasticsearch bulk request payload using java high level rest client and bulk processor? I have tried below ```auto requestConfigBuilder.setContentCompressionEnabled(true); ``` But it does not seem to be compressing the request payload. Do I need to enable any other settings? Any inputs? ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [August 4, 2021, 1:35pm UTC](https://discuss.elastic.co/t/compress-elasticsearch-bulkrequest-using-java-high-level-rest-client-and-bulk-processor/280425/2 "2021-08-04T13:35:23Z")
Hey, any chance to create a code snippet that can be tested in an IDE and follow the steps from your setup? Thank you! ---
**Author:** ![Praveen\_Atluri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/praveen_atluri/32/39721_2.png) [@Praveen\_Atluri](https://discuss.elastic.co/u/Praveen_Atluri)\ **Post date:** [August 4, 2021, 2:41pm UTC](https://discuss.elastic.co/t/compress-elasticsearch-bulkrequest-using-java-high-level-rest-client-and-bulk-processor/280425/3 "2021-08-04T14:41:43Z")
```auto RestClientBuilder restClientBuilder = RestClient.builder(httpHosts); SniffOnFailureListener sniffOnFailureListener = new SniffOnFailureListener(); if (!config.isEsNodeSniffingDisabled()) { restClientBuilder.setFailureListener(sniffOnFailureListener); } restClientBuilder.setHttpClientConfigCallback(httpClientBuilder -> { httpClientBuilder .setDefaultConnectionConfig(ConnectionConfig.custom().setCharset(StandardCharsets.UTF_8).build()); return httpClientBuilder; }); restClientBuilder.setRequestConfigCallback(requestConfigBuilder -> { requestConfigBuilder.setSocketTimeout(config.getEsSocketTimeout()*1000); requestConfigBuilder.setContentCompressionEnabled(true); requestConfigBuilder.setConnectTimeout(config.getEsConnectionTimeout()*1000); requestConfigBuilder.setConnectionRequestTimeout(config.getEsConnectionTimeout()*1000); return requestConfigBuilder; }); return new RestHighLevelClient(restClientBuilder); ``` ---
**Author:** ![Praveen\_Atluri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/praveen_atluri/32/39721_2.png) [@Praveen\_Atluri](https://discuss.elastic.co/u/Praveen_Atluri)\ **Post date:** [August 6, 2021, 6:05am UTC](https://discuss.elastic.co/t/compress-elasticsearch-bulkrequest-using-java-high-level-rest-client-and-bulk-processor/280425/4 "2021-08-06T06:05:51Z")
Just to give more context, We were trying to do load test on es cluster (with 150 data nodes) deployed on AWS EC2 instances. We were not able to get the desired ingestion throughput despite making all the recommended settings for better indexing performance. Then we saw that the problem was with the network bandwidth. The AWS i3.2x large instances come with Up to 10 Gigabit/second network bandwidth and this network was clogged. So we were not able to ingest the data at a better throughput. So, I was thinking we could improve the throughput by compressing the bulk payload. I could not find any documentation on how to compress the payload using java high level rest client . Any inputs would be really helpful. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 6, 2021, 11:42am UTC](https://discuss.elastic.co/t/compress-elasticsearch-bulkrequest-using-java-high-level-rest-client-and-bulk-processor/280425/5 "2021-08-06T11:42:52Z")
> [@Praveen\_Atluri](#): > > `requestConfigBuilder.setContentCompressionEnabled(true);` I believe that the only effect of this line is to indicate to Elasticsearch that the client will accept a compressed _response_. If you want the client to compress the requests too you should call `RestClientBuilder#setCompressionEnabled` instead. Note that compression takes substantial extra CPU effort. An `i3.2xlarge` only has 8 CPUs, I think you'd need several times that number to reach 10Gbps of throughput. ---
**Author:** ![Praveen\_Atluri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/praveen_atluri/32/39721_2.png) [@Praveen\_Atluri](https://discuss.elastic.co/u/Praveen_Atluri)\ **Post date:** [August 6, 2021, 12:05pm UTC](https://discuss.elastic.co/t/compress-elasticsearch-bulkrequest-using-java-high-level-rest-client-and-bulk-processor/280425/6 "2021-08-06T12:05:09Z")
Thank you @DavidTurner , Let me try this and see if I see any improvement in throughput. Thank you! ---
**Author:** ![Praveen\_Atluri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/praveen_atluri/32/39721_2.png) [@Praveen\_Atluri](https://discuss.elastic.co/u/Praveen_Atluri)\ **Post date:** [August 6, 2021, 8:43pm UTC](https://discuss.elastic.co/t/compress-elasticsearch-bulkrequest-using-java-high-level-rest-client-and-bulk-processor/280425/7 "2021-08-06T20:43:06Z")
@DavidTurner , I see that this method ```auto RestClientBuilder#setCompressionEnabled ``` is available only from elasticsearch-rest-client version 7.10.1. But we are running elasticsearch server version 7.8.0. Is it possible to use rest client version 7.10.1 to compress bulk request and ingest into elasticsearch server 7.8.0? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 6, 2021, 9:32pm UTC](https://discuss.elastic.co/t/compress-elasticsearch-bulkrequest-using-java-high-level-rest-client-and-bulk-processor/280425/8 "2021-08-06T21:32:36Z")
Ah yes, it was introduced in 7.10.0 (see below). I don't know another way to do it with the high-level REST client, although of course it's pretty simple to use a bare HTTP client to send bulk requests with compression. > > > Adds a \`RestClient.setCompressionEnabled()\` setting that will gzip- > compress re…quest bodies and add a \`Accept-Encoding: gzip\` header so > that the ES server can send compressed responses. > > Fixes #62044 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 3, 2021, 9:32pm UTC](https://discuss.elastic.co/t/compress-elasticsearch-bulkrequest-using-java-high-level-rest-client-and-bulk-processor/280425/9 "2021-09-03T21:32:41Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.