Enable HTTP response compression when reading from ES

I am surprised to see there is no compression when reading data from elasticsearch server ?

Using a proxy, I can see the request:

User-Agent	Jakarta Commons-HttpClient/3.1
Host	es.analytics
Content-Length	68
Content-Type	application/json; charset=UTF-8

Since Hadoop is big-data, should we enable it by default?

@ebuildy This is an issue with our version of the HTTP client that is provided with Hadoop. To shed some more light on it from this issue:

ES-Hadoop is built on top of Apache Commons HTTP v3.0.1 which does not support compression. Experimental support for compression was added in v4.1 and enabled by default in v4.3, but will not be backported to 3 in any capacity. Since we use the library versions that are provided with Hadoop to avoid conflicting jars and to facilitate simpler usage of the library this would need to wait until Hadoop updates their version of Commons HTTP.

Yeah this makes sense, hopefully Hadoop v3 will provide version 4.5


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.