Hive 3.1.1 ES-Hadoop Connector

Elastic Stack Elasticsearch
1

Continuing the discussion from Support for HDFS 3.1.1 and Hive 3.1:

My Hive version is 3.1.1

Hi I keep getting error

Error: java.lang.RuntimeException: Error in configuring object
	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1911)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
	... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
	at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
	... 14 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
	... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:137)
	... 22 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Unsupported/Unknown Elasticsearch version 7.3.1
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSinkOperator.java:623)
	at org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator.initializeOp(VectorFileSinkOperator.java:84)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:573)
	at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:525)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:386)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:573)
	at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:525)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:386)
	at org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.initializeMapOperator(VectorMapOperator.java:591)
	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:116)
	... 22 more
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Unsupported/Unknown Elasticsearch version 7.3.1
	at org.elasticsearch.hadoop.util.EsMajorVersion.parse(EsMajorVersion.java:82)
	at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:314)
	at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:87)
	at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:102)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSinkOperator.java:532)
	... 32 more


19/10/31 15:22:41 INFO impl.YarnClientImpl: Killed application application_1571499978896_0102
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

when I try to insert data. Any idea the reasons? Is Hive 3.1.1 supported yet?

2

Unsupported/Unknown Elasticsearch version 7.3.1

This looks like ES-Hadoop is not familiar with the version of Elasticsearch it is connecting to. I would check that your ES-Hadoop version is greater than or equal to the Elasticsearch version you are reading and try again.

That said, Hive 3.1.1 is not yet supported. We're hoping to get support for it soon, and we appreciate the community's patience on the matter.

3

Hi @Loo_Ying_Ting.

please try to add below jars file , i hope this will work.

add jar /home/cdh/BigData/Hive/Data/elasticsearch-hadoop-7.1.1.jar;
add jar /home/cdh/BigData/Hive/Data/elasticsearch-hadoop-hive-7.1.1.jar;
add jar /home/cdh/BigData/Hive/Data/commons-httpclient-3.0.1.jar;

Thanks
HadoopHelp

4

above will support and work perfectly !!

please @Loo_Ying_Ting reply here if you having the same issue.

Thanks
HadoopHelp

5

Please only add one or the other. Adding both of these may trip up ES-Hadoop's Jar Hell detection when it loads the library classes.

6

Hi @james.baiera.

you are correct.we can select any one i shared all possible jars.

Thanks
HadoopHelp

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.