Hi,
I am having a hard time integrating the High-level-REST (6.4) client into a MapReduce job
Looks like a Jar dependency problem. I tried to shade - no luck at all.
The problem seems to be in the httpcore libs from Apache.
What i find strange is that the exception is thrown in in org.apache class, and i have shaded all org.apache.http to hidden.org.apache.http
Any help is appreciated.
Hadoop version : Hadoop 2.6.0-cdh5.7.1
ES version : 6.4
Here is the stacktrace:
2018-11-09 16:54:10,086 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchFieldError: INSTANCE at org.apache.http.impl.nio.codecs.DefaultHttpRequestWriterFactory.<init>(DefaultHttpRequestWriterFactory.java:53) at org.apache.http.impl.nio.codecs.DefaultHttpRequestWriterFactory.<init>(DefaultHttpRequestWriterFactory.java:57) at org.apache.http.impl.nio.codecs.DefaultHttpRequestWriterFactory.<clinit>(DefaultHttpRequestWriterFactory.java:47) at org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory.<init>(ManagedNHttpClientConnectionFactory.java:75) at org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory.<init>(ManagedNHttpClientConnectionFactory.java:83) at org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory.<clinit>(ManagedNHttpClientConnectionFactory.java:64) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$InternalConnectionFactory.<init>(PoolingNHttpClientConnectionManager.java:553) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.<init>(PoolingNHttpClientConnectionManager.java:163) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.<init>(PoolingNHttpClientConnectionManager.java:147) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.<init>(PoolingNHttpClientConnectionManager.java:119) at org.apache.http.impl.nio.client.HttpAsyncClientBuilder.build(HttpAsyncClientBuilder.java:668) at org.elasticsearch.client.RestClientBuilder$2.run(RestClientBuilder.java:230) at org.elasticsearch.client.RestClientBuilder$2.run(RestClientBuilder.java:227) at java.security.AccessController.doPrivileged(Native Method) at org.elasticsearch.client.RestClientBuilder.createHttpClient(RestClientBuilder.java:227) at org.elasticsearch.client.RestClientBuilder.access$000(RestClientBuilder.java:42) at org.elasticsearch.client.RestClientBuilder$1.run(RestClientBuilder.java:198) at org.elasticsearch.client.RestClientBuilder$1.run(RestClientBuilder.java:195) at java.security.AccessController.doPrivileged(Native Method) at org.elasticsearch.client.RestClientBuilder.build(RestClientBuilder.java:195) at org.elasticsearch.client.RestHighLevelClient.<init>(RestHighLevelClient.java:221) at org.elasticsearch.client.RestHighLevelClient.<init>(RestHighLevelClient.java:213) at com.vesseltracker.hadoop.mapreduce.ReduceTemplate.getElasticSearchClient(ReduceTemplate.java:156) at com.vesseltracker.hadoop.mapreduce.ReduceTemplate.setup(ReduceTemplate.java:122) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:168) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)