ElasticSearch-Hadoop:Loading data into Elasticsearch through hive querl showing DDLTask error


(mohit.Kumar) #1

hi folks,
I facing following error while load data into elasticsearch using hive
query.
ERROR:-
14/08/30 02:05:04 INFO log.PerfLogger:
14/08/30 02:05:04 INFO ql.Driver: Starting command: CREATE EXTERNAL TABLE
eslogs (time STRING, extension STRING, clientip STRING, request STRING,
response INT, agent STRING)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'demo/hive','es.nodes' =
'http://192.168.186.1','es.port'
= '9200')
14/08/30 02:05:04 INFO impl.TimelineClientImpl: Timeline service address:
http://sandbox.hortonworks.com:8188/ws/v1/timeline/
14/08/30 02:05:04 INFO hooks.ATSHook: Created ATS Hook
14/08/30 02:05:04 INFO log.PerfLogger:
14/08/30 02:05:04 INFO log.PerfLogger: </PERFLOG
method=PreHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1409389504583
end=1409389504583 duration=0 from=org.apache.hadoop.hive.ql.Driver>
14/08/30 02:05:04 INFO log.PerfLogger: </PERFLOG method=TimeToSubmit
start=1409389502353 end=1409389504583 duration=2230
from=org.apache.hadoop.hive.ql.Driver>
14/08/30 02:05:04 INFO log.PerfLogger:
14/08/30 02:05:04 INFO log.PerfLogger:
14/08/30 02:05:04 INFO exec.DDLTask: Use StorageHandler-supplied
org.elasticsearch.hadoop.hive.EsSerDe for table eslogs
14/08/30 02:05:04 ERROR exec.DDLTask: java.lang.IllegalAccessError:
org/elasticsearch/hadoop/hive/HiveUtils
at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:80)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:339)
at
org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:283)
at
org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:276)
at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:626)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:593)
at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4194)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:281)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
at
com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.execute(BeeswaxServiceImpl.java:349)
at
com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:614)
at
com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:603)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:356)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1537)
at
com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1.run(BeeswaxServiceImpl.java:603)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

14/08/30 02:05:04 INFO impl.TimelineClientImpl: Timeline service address:
http://sandbox.hortonworks.com:8188/ws/v1/timeline/
14/08/30 02:05:04 INFO hooks.ATSHook: Created ATS Hook
14/08/30 02:05:04 INFO log.PerfLogger:
14/08/30 02:05:04 INFO log.PerfLogger: </PERFLOG
method=FailureHook.org.apache.hadoop.hive.ql.hooks.ATSHook
start=1409389504698 end=1409389504698 duration=0
from=org.apache.hadoop.hive.ql.Driver>
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask.
org/elasticsearch/hadoop/hive/HiveUtils
14/08/30 02:05:04 ERROR ql.Driver: FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.DDLTask.
org/elasticsearch/hadoop/hive/HiveUtils
14/08/30 02:05:04 INFO log.PerfLogger: </PERFLOG method=Driver.execute
start=1409389504462 end=1409389504700 duration=238
from=org.apache.hadoop.hive.ql.Driver>
14/08/30 02:05:04 ERROR beeswax.BeeswaxServiceImpl: Exception while
processing query
BeeswaxException(message:Driver returned: 1. Errors: OK
converting to local hdfs://
sandbox.hortonworks.com:8020/user/hue/elasticsearch-hadoop-2.1.0.Beta1.jar
Added
/tmp/a1c5d5b9-3aa7-41f8-8e93-73f8277a9841_resources/elasticsearch-hadoop-2.1.0.Beta1.jar
to class path
Added resource:
/tmp/a1c5d5b9-3aa7-41f8-8e93-73f8277a9841_resources/elasticsearch-hadoop-2.1.0.Beta1.jar
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask.
org/elasticsearch/hadoop/hive/HiveUtils
, log_context:d0960413-bfb9-4bdf-b615-26ad7821587b,
handle:QueryHandle(id:d0960413-bfb9-4bdf-b615-26ad7821587b,
log_context:d0960413-bfb9-4bdf-b615-26ad7821587b), SQLState: )
at
com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.execute(BeeswaxServiceImpl.java:356)
at
com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:614)
at
com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:603)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:356)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1537)
at
com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1.run(BeeswaxServiceImpl.java:603)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

My setup:-

  1. Hortonworks Sandbox VM (running on vmware player):-
    Hortonworks_Sandbox_2.1
  2. elasticsearch up & running :- Elasticsearch version 1.3.1
  3. elasticsearch-hadoop jars :- elasticsearch-hadoop-2.1.0.Beta1,
    elasticsearch-hadoop-1.3.0.M1

I am following this url :-
https://github.com/hortonworks/hadoop-tutorials/blob/master/Community/T07_Elasticsearch_Hadoop_Integration.md

Hive Queries:-
Create table in hive and load data from zip file:-

CREATE TABLE apachelog (type STRING, time STRING, ext STRING, ip STRING,
req STRING, res INT, bytes INT, phpmem INT, agent STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';
LOAD DATA INPATH '/user/hue/apache/apache.log' OVERWRITE INTO TABLE
apachelog;

Load data into easlticsearch query:-

CREATE EXTERNAL TABLE eslogs (time STRING, extension STRING, clientip
STRING, request STRING, response INT, agent STRING)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'demo/hive','es.nodes' =
'http://192.168.186.1','es.port'
= '9200');

INSERT OVERWRITE TABLE eslogs SELECT s.time, s.ext, s.ip, s.req, s.res,
s.agent FROM apachelog s;

Kindly suggest me where I am doing wrong and what is the meaning of this
error and why it is coming?

Regrads
Mohit Kumar Yadav
(MCA/BBA)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAK6eDKeY0%2B5r_WTh8mpFJiXJOMC2P__G254XZQDsXmJczHMtMQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Costin Leau) #2

It looks like there's a classpath issue (notice the HiveUtils error there). Most likely because you have two versions
of es-hadoop in your classpath (2.1.0.Beta1 and 1.3.0.M1).

Use only one - I suggest 2.1.0.Beta1.

Cheers,

On 9/5/14 3:39 PM, Mohit Kumar Yadav wrote:

hi folks,
I facing following error while load data into elasticsearch using hive query.
ERROR:-
14/08/30 02:05:04 INFO log.PerfLogger:
14/08/30 02:05:04 INFO ql.Driver: Starting command: CREATE EXTERNAL TABLE eslogs (time STRING, extension STRING,
clientip STRING, request STRING, response INT, agent STRING)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'demo/hive','es.nodes' = 'http://192.168.186.1','es.port' = '9200')
14/08/30 02:05:04 INFO impl.TimelineClientImpl: Timeline service address:
http://sandbox.hortonworks.com:8188/ws/v1/timeline/
14/08/30 02:05:04 INFO hooks.ATSHook: Created ATS Hook
14/08/30 02:05:04 INFO log.PerfLogger:
14/08/30 02:05:04 INFO log.PerfLogger: </PERFLOG method=PreHook.org.apache.hadoop.hive.ql.hooks.ATSHook
start=1409389504583 end=1409389504583 duration=0 from=org.apache.hadoop.hive.ql.Driver>
14/08/30 02:05:04 INFO log.PerfLogger: </PERFLOG method=TimeToSubmit start=1409389502353 end=1409389504583 duration=2230
from=org.apache.hadoop.hive.ql.Driver>
14/08/30 02:05:04 INFO log.PerfLogger:
14/08/30 02:05:04 INFO log.PerfLogger:
14/08/30 02:05:04 INFO exec.DDLTask: Use StorageHandler-supplied org.elasticsearch.hadoop.hive.EsSerDe for table eslogs
14/08/30 02:05:04 ERROR exec.DDLTask: java.lang.IllegalAccessError: org/elasticsearch/hadoop/hive/HiveUtils
at org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:80)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:339)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:283)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:276)
at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:626)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:593)
at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4194)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:281)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.execute(BeeswaxServiceImpl.java:349)
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:614)
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:603)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:356)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1537)
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1.run(BeeswaxServiceImpl.java:603)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

14/08/30 02:05:04 INFO impl.TimelineClientImpl: Timeline service address:
http://sandbox.hortonworks.com:8188/ws/v1/timeline/
14/08/30 02:05:04 INFO hooks.ATSHook: Created ATS Hook
14/08/30 02:05:04 INFO log.PerfLogger:
14/08/30 02:05:04 INFO log.PerfLogger: </PERFLOG method=FailureHook.org.apache.hadoop.hive.ql.hooks.ATSHook
start=1409389504698 end=1409389504698 duration=0 from=org.apache.hadoop.hive.ql.Driver>
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org/elasticsearch/hadoop/hive/HiveUtils
14/08/30 02:05:04 ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
org/elasticsearch/hadoop/hive/HiveUtils
14/08/30 02:05:04 INFO log.PerfLogger: </PERFLOG method=Driver.execute start=1409389504462 end=1409389504700
duration=238 from=org.apache.hadoop.hive.ql.Driver>
14/08/30 02:05:04 ERROR beeswax.BeeswaxServiceImpl: Exception while processing query
BeeswaxException(message:Driver returned: 1. Errors: OK
converting to local hdfs://sandbox.hortonworks.com:8020/user/hue/elasticsearch-hadoop-2.1.0.Beta1.jar
http://sandbox.hortonworks.com:8020/user/hue/elasticsearch-hadoop-2.1.0.Beta1.jar
Added /tmp/a1c5d5b9-3aa7-41f8-8e93-73f8277a9841_resources/elasticsearch-hadoop-2.1.0.Beta1.jar to class path
Added resource: /tmp/a1c5d5b9-3aa7-41f8-8e93-73f8277a9841_resources/elasticsearch-hadoop-2.1.0.Beta1.jar
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org/elasticsearch/hadoop/hive/HiveUtils
, log_context:d0960413-bfb9-4bdf-b615-26ad7821587b, handle:QueryHandle(id:d0960413-bfb9-4bdf-b615-26ad7821587b,
log_context:d0960413-bfb9-4bdf-b615-26ad7821587b), SQLState: )
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.execute(BeeswaxServiceImpl.java:356)
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:614)
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:603)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:356)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1537)
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1.run(BeeswaxServiceImpl.java:603)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

My setup:-

  1. Hortonworks Sandbox VM (running on vmware player):- Hortonworks_Sandbox_2.1
  2. elasticsearch up & running :- Elasticsearch version 1.3.1
  3. elasticsearch-hadoop jars :- elasticsearch-hadoop-2.1.0.Beta1,elasticsearch-hadoop-1.3.0.M1

I am following this url :-
https://github.com/hortonworks/hadoop-tutorials/blob/master/Community/T07_Elasticsearch_Hadoop_Integration.md

Hive Queries:-
Create table in hive and load data from zip file:-

CREATE TABLE apachelog (type STRING, time STRING, ext STRING, ip STRING, req STRING, res INT, bytes INT, phpmem INT,
agent STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';
LOAD DATA INPATH '/user/hue/apache/apache.log' OVERWRITE INTO TABLE apachelog;

Load data into easlticsearch query:-

CREATE EXTERNAL TABLE eslogs (time STRING, extension STRING, clientip STRING, request STRING, response INT, agent STRING)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'demo/hive','es.nodes' = 'http://192.168.186.1','es.port' = '9200');

INSERT OVERWRITE TABLE eslogs SELECT s.time, s.ext, s.ip, s.req, s.res, s.agent FROM apachelog s;

Kindly suggest me where I am doing wrong and what is the meaning of this error and why it is coming?

Regrads
Mohit Kumar Yadav
(MCA/BBA)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAK6eDKeY0%2B5r_WTh8mpFJiXJOMC2P__G254XZQDsXmJczHMtMQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAK6eDKeY0%2B5r_WTh8mpFJiXJOMC2P__G254XZQDsXmJczHMtMQ%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5409AFE5.9010904%40gmail.com.
For more options, visit https://groups.google.com/d/optout.


(mohit.Kumar) #3

Thanks Costin for replying,

That works for me but now I am facing with another issue.

data are not loading into elasticsearch. whenever I fires insertion query
it shows following error.

Query ID = hue_20140908052626_c15ea214-c656-47b1-9132-49d3c407b8f8
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1410175571220_0003, Tracking URL = http://sandbox.hortonworks.com:8088/proxy/application_1410175571220_0003/
Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1410175571220_0003
Hadoop job information for Stage-0: number of mappers: 1; number of reducers: 0
2014-09-08 05:26:57,048 Stage-0 map = 0%, reduce = 0%
2014-09-08 05:27:29,019 Stage-0 map = 100%, reduce = 0%
Ended Job = job_1410175571220_0003 with errors
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
, log_context:d3f5bc11-575d-4dd5-91a4-c6d9c0032c25, handle:QueryHandle(id:d3f5bc11-575d-4dd5-91a4-c6d9c0032c25, log_context:d3f5bc11-575d-4dd5-91a4-c6d9c0032c25), SQLState: )
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.execute(BeeswaxServiceImpl.java:356)
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:614)
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:603)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:356)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1537)
at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1.run(BeeswaxServiceImpl.java:603)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Hive Queries:-Create table in hive and load data from zip file:-CREATE TABLE apachelog (type STRING, time STRING, ext STRING, ip STRING, req STRING, res INT, bytes INT, phpmem INT, agent STRING)ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';LOAD DATA INPATH '/user/hue/apache/apache.log' OVERWRITE INTO TABLE apachelog;
Load data into easlticsearch query:-CREATE EXTERNAL TABLE eslogs (time STRING, extension STRING, clientip STRING, request STRING, response INT, agent STRING)STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'TBLPROPERTIES('es.resource' = 'demo/hive','es.nodes' = 'http://192.168.186.1','es.port' = '9200');
INSERT OVERWRITE TABLE eslogs SELECT s.time, s.ext, s.ip, s.req, s.res, s.agent FROM apachelog s;
please suggest me what should i do to remove this error.

thanks in advance

On Friday, September 5, 2014 6:13:58 PM UTC+5:30, Costin Leau wrote:

It looks like there's a classpath issue (notice the HiveUtils error
there). Most likely because you have two versions
of es-hadoop in your classpath (2.1.0.Beta1 and 1.3.0.M1).

Use only one - I suggest 2.1.0.Beta1.

Cheers,

On 9/5/14 3:39 PM, Mohit Kumar Yadav wrote:

hi folks,
I facing following error while load data into elasticsearch using hive
query.
ERROR:-
14/08/30 02:05:04 INFO log.PerfLogger:
14/08/30 02:05:04 INFO ql.Driver: Starting command: CREATE EXTERNAL
TABLE eslogs (time STRING, extension STRING,
clientip STRING, request STRING, response INT, agent STRING)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'demo/hive','es.nodes' = '
http://192.168.186.1','es.port' = '9200')
14/08/30 02:05:04 INFO impl.TimelineClientImpl: Timeline service
address:
http://sandbox.hortonworks.com:8188/ws/v1/timeline/
14/08/30 02:05:04 INFO hooks.ATSHook: Created ATS Hook
14/08/30 02:05:04 INFO log.PerfLogger:
14/08/30 02:05:04 INFO log.PerfLogger: </PERFLOG
method=PreHook.org.apache.hadoop.hive.ql.hooks.ATSHook
start=1409389504583 end=1409389504583 duration=0
from=org.apache.hadoop.hive.ql.Driver>
14/08/30 02:05:04 INFO log.PerfLogger: </PERFLOG method=TimeToSubmit
start=1409389502353 end=1409389504583 duration=2230
from=org.apache.hadoop.hive.ql.Driver>
14/08/30 02:05:04 INFO log.PerfLogger:
14/08/30 02:05:04 INFO log.PerfLogger:
14/08/30 02:05:04 INFO exec.DDLTask: Use StorageHandler-supplied
org.elasticsearch.hadoop.hive.EsSerDe for table eslogs
14/08/30 02:05:04 ERROR exec.DDLTask: java.lang.IllegalAccessError:
org/elasticsearch/hadoop/hive/HiveUtils
at
org.elasticsearch.hadoop.hive.EsSerDe.initialize(EsSerDe.java:80)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:339)

 at 

org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:283)

 at 

org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:276)

 at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:626) 
 at 

org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:593)

 at 

org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4194)

 at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:281) 
 at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153) 
 at 

org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)

 at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504) 
 at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271) 
 at 

com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.execute(BeeswaxServiceImpl.java:349)

 at 

com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:614)

 at 

com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:603)

 at java.security.AccessController.doPrivileged(Native Method) 
 at javax.security.auth.Subject.doAs(Subject.java:356) 
 at 

org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1537)

 at 

com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1.run(BeeswaxServiceImpl.java:603)

 at 

java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

 at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
 at 

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

 at 

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

 at java.lang.Thread.run(Thread.java:744) 

14/08/30 02:05:04 INFO impl.TimelineClientImpl: Timeline service
address:
http://sandbox.hortonworks.com:8188/ws/v1/timeline/
14/08/30 02:05:04 INFO hooks.ATSHook: Created ATS Hook
14/08/30 02:05:04 INFO log.PerfLogger:
14/08/30 02:05:04 INFO log.PerfLogger: </PERFLOG
method=FailureHook.org.apache.hadoop.hive.ql.hooks.ATSHook
start=1409389504698 end=1409389504698 duration=0
from=org.apache.hadoop.hive.ql.Driver>
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask.
org/elasticsearch/hadoop/hive/HiveUtils
14/08/30 02:05:04 ERROR ql.Driver: FAILED: Execution Error, return code
1 from org.apache.hadoop.hive.ql.exec.DDLTask.
org/elasticsearch/hadoop/hive/HiveUtils
14/08/30 02:05:04 INFO log.PerfLogger: </PERFLOG method=Driver.execute
start=1409389504462 end=1409389504700
duration=238 from=org.apache.hadoop.hive.ql.Driver>
14/08/30 02:05:04 ERROR beeswax.BeeswaxServiceImpl: Exception while
processing query
BeeswaxException(message:Driver returned: 1. Errors: OK
converting to local hdfs://
sandbox.hortonworks.com:8020/user/hue/elasticsearch-hadoop-2.1.0.Beta1.jar
<
http://sandbox.hortonworks.com:8020/user/hue/elasticsearch-hadoop-2.1.0.Beta1.jar>

Added
/tmp/a1c5d5b9-3aa7-41f8-8e93-73f8277a9841_resources/elasticsearch-hadoop-2.1.0.Beta1.jar
to class path
Added resource:
/tmp/a1c5d5b9-3aa7-41f8-8e93-73f8277a9841_resources/elasticsearch-hadoop-2.1.0.Beta1.jar

FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask.
org/elasticsearch/hadoop/hive/HiveUtils
, log_context:d0960413-bfb9-4bdf-b615-26ad7821587b,
handle:QueryHandle(id:d0960413-bfb9-4bdf-b615-26ad7821587b,
log_context:d0960413-bfb9-4bdf-b615-26ad7821587b), SQLState: )
at
com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.execute(BeeswaxServiceImpl.java:356)

 at 

com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:614)

 at 

com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:603)

 at java.security.AccessController.doPrivileged(Native Method) 
 at javax.security.auth.Subject.doAs(Subject.java:356) 
 at 

org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1537)

 at 

com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1.run(BeeswaxServiceImpl.java:603)

 at 

java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

 at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
 at 

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

 at 

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

 at java.lang.Thread.run(Thread.java:744) 

My setup:-

  1. Hortonworks Sandbox VM (running on vmware player):-
    Hortonworks_Sandbox_2.1
  2. elasticsearch up & running :- Elasticsearch version 1.3.1
  3. elasticsearch-hadoop jars :-
    elasticsearch-hadoop-2.1.0.Beta1,elasticsearch-hadoop-1.3.0.M1

I am following this url :-

https://github.com/hortonworks/hadoop-tutorials/blob/master/Community/T07_Elasticsearch_Hadoop_Integration.md

Hive Queries:-
Create table in hive and load data from zip file:-

CREATE TABLE apachelog (type STRING, time STRING, ext STRING, ip STRING,
req STRING, res INT, bytes INT, phpmem INT,
agent STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';
LOAD DATA INPATH '/user/hue/apache/apache.log' OVERWRITE INTO TABLE
apachelog;

Load data into easlticsearch query:-

CREATE EXTERNAL TABLE eslogs (time STRING, extension STRING, clientip
STRING, request STRING, response INT, agent STRING)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'demo/hive','es.nodes' = '
http://192.168.186.1','es.port' = '9200');

INSERT OVERWRITE TABLE eslogs SELECT s.time, s.ext, s.ip, s.req, s.res,
s.agent FROM apachelog s;

Kindly suggest me where I am doing wrong and what is the meaning of this
error and why it is coming?

Regrads
Mohit Kumar Yadav
(MCA/BBA)

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to
elasticsearc...@googlegroups.com <javascript:> <mailto:
elasticsearch+unsubscribe@googlegroups.com <javascript:>>.
To view this discussion on the web visit

https://groups.google.com/d/msgid/elasticsearch/CAK6eDKeY0%2B5r_WTh8mpFJiXJOMC2P__G254XZQDsXmJczHMtMQ%40mail.gmail.com

<
https://groups.google.com/d/msgid/elasticsearch/CAK6eDKeY0%2B5r_WTh8mpFJiXJOMC2P__G254XZQDsXmJczHMtMQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.

For more options, visit https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/440d87a5-47f4-4ff9-944a-dd01f5b367eb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #4