I am surprised to see there is no compression when reading data from elasticsearch server ?
Using a proxy, I can see the request:
User-Agent Jakarta Commons-HttpClient/3.1
Host es.analytics
Content-Length 68
Content-Type application/json; charset=UTF-8
Since Hadoop is big-data, should we enable it by default?
@ebuildy This is an issue with our version of the HTTP client that is provided with Hadoop. To shed some more light on it from this issue:
ES-Hadoop is built on top of Apache Commons HTTP v3.0.1 which does not support compression. Experimental support for compression was added in v4.1 and enabled by default in v4.3, but will not be backported to 3 in any capacity. Since we use the library versions that are provided with Hadoop to avoid conflicting jars and to facilitate simpler usage of the library this would need to wait until Hadoop updates their version of Commons HTTP.
Yeah this makes sense, hopefully Hadoop v3 will provide version 4.5
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.