Elasticsearch for Hadoop issue

Elastic Stack Elasticsearch
1

I had fixed the issue of inserting into the Elasticsearch table by copying the
elasticsearch-hadoop-5.5.0.jar to the the $HIVE_HOME/lib folder but now I am having the same issue while selecting from the table , what could be the issue ?

0: jdbc:hive2://hadoop2:10000/default> list jar;
+------------------------------------------------------------------------------------------+--+
| /tmp/99fbb7fd-d0e8-4bef-8d28-761004cd559d_resources/elasticsearch-hadoop-5.5.0.jar |
| /usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar |
| /tmp/99fbb7fd-d0e8-4bef-8d28-761004cd559d_resources/elasticsearch-hadoop-hive-5.5.1.jar |
+------------------------------------------------------------------------------------------+--+
3 rows selected (0.004 seconds)
0: jdbc:hive2://hadoop2:10000/default>
0: jdbc:hive2://hadoop2:10000/default>
0: jdbc:hive2://hadoop2:10000/default> select txn_process_date from pa_lane_txn_es;
Error: java.io.IOException: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Unsupported/Unknown Elasticsearch version 5.5.0 (state=,code=0)
0: jdbc:hive2://hadoop2:10000/default>
0: jdbc:hive2://hadoop2:10000/default>

2

@aliyesami I would avoid adding multiple versions of ES-Hadoop to your classpath. There is code that checks that there are not multiple versions of the library on the classpath and will throw exceptions if multiple are detected. I would use only one of the ES-Hadoop jars at a time.

3

still the same issue . I only have one jar registered

[root@hadoop1 ~]# echo $CLASSPATH
/usr/hdp/2.5.3.0-37/flume/lib/*
[root@hadoop1 ~]#
[root@hadoop1 ~]# beeline -u 'jdbc:hive2://hadoop2:10000/default;principal=hive/hadoop2@TOLLS.DOT.STATE.FL.US'
Connecting to jdbc:hive2://hadoop2:10000/default;principal=hive/hadoop2@TOLLS.DOT.STATE.FL.US
17/07/31 07:36:06 [TGT Renewer for sami@TOLLS.DOT.STATE.FL.US]: WARN security.UserGroupInformation: Exception encountered while running the renewal command. Aborting renew thread. ExitCodeException exitCode=1: kinit: Ticket expired while renewing credentials

Connected to: Apache Hive (version 1.2.1000.2.5.3.0-37)
Driver: Hive JDBC (version 1.2.1000.2.5.3.0-37)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.1000.2.5.3.0-37 by Apache Hive
0: jdbc:hive2://hadoop2:10000/default> add JAR hdfs:///tmp/elasticsearch-hadoop-5.5.0.jar;
INFO : converting to local hdfs:///tmp/elasticsearch-hadoop-5.5.0.jar
INFO : Added [/tmp/7940dec9-b87c-4ed6-8cf3-b4d8e73ab8d0_resources/elasticsearch-hadoop-5.5.0.jar] to class path
INFO : Added resources: [hdfs:///tmp/elasticsearch-hadoop-5.5.0.jar]
No rows affected (0.094 seconds)
0: jdbc:hive2://hadoop2:10000/default> add jar /usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar;
INFO : Added [/usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar] to class path
INFO : Added resources: [/usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar]
No rows affected (0.002 seconds)
0: jdbc:hive2://hadoop2:10000/default> select * from pa_lane_txn_es limit 10;
Error: java.io.IOException: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Unsupported/Unknown Elasticsearch version 5.5.0 (state=,code=0)
0: jdbc:hive2://hadoop2:10000/default> QUIT
0: jdbc:hive2://hadoop2:10000/default> [root@hadoop1 ~]#
[root@hadoop1 ~]# ls /usr/hdp/2.5.3.0-37/hive/lib/elasticsearch-hadoop*
/usr/hdp/2.5.3.0-37/hive/lib/elasticsearch-hadoop-5.5.0.jar
[root@hadoop1 ~]#
[root@hadoop1 ~

4

also if its a jar issue why am I able to insert into this elastic table fine and it populates the index properly but just can't select from it?

5

if I connect as hive and not beeline , I get slightly different error

hive> list jar;
hive> add JAR hdfs:///tmp/elasticsearch-hadoop-5.5.0.jar;
converting to local hdfs:///tmp/elasticsearch-hadoop-5.5.0.jar
Added [/tmp/71fb60b9-b319-4edf-b8ae-cba6fc675e1c_resources/elasticsearch-hadoop-5.5.0.jar] to class path
Added resources: [hdfs:///tmp/elasticsearch-hadoop-5.5.0.jar]
hive> add jar /usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar;
Added [/usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar] to class path
Added resources: [/usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar]
hive> list jar;
/usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar
/tmp/71fb60b9-b319-4edf-b8ae-cba6fc675e1c_resources/elasticsearch-hadoop-5.5.0.jar
hive>
> select * from pa_lane_txn_es limit 10;
FAILED: EsHadoopIllegalArgumentException Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
hive>

6

ok this issue is resolved by using hive and having just one jar registered, beeline still is giving error but I can continue .
thanks for your help

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.