# Trouble to load data from my hadoop cluster to elasticsearch via pig and hive
**URL:** https://discuss.elastic.co/t/trouble-to-load-data-from-my-hadoop-cluster-to-elasticsearch-via-pig-and-hive/22580
**Category:** Elasticsearch
**Created:** [March 9, 2015, 8:57am UTC](https://discuss.elastic.co/t/trouble-to-load-data-from-my-hadoop-cluster-to-elasticsearch-via-pig-and-hive/22580 "2015-03-09T08:57:33Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@omar\_ben\_salem](https://discuss.elastic.co/u/omar_ben_salem)
#### Post date: [March 9, 2015, 8:57am UTC](https://discuss.elastic.co/t/trouble-to-load-data-from-my-hadoop-cluster-to-elasticsearch-via-pig-and-hive/22580/1 "2015-03-09T08:57:33Z")
I have some issues with my project and here I am seeking for some guidance
and help
here is the situation :
Pig :
I have a hadoop cluster managed by Cloudera CDH 5.3.
I have ElasticSearch 1.4.4 installed in my master machine(10.44.162.169)
I have downloaded the marvel plugin and so access to my ES via :
[http://10.44.162.169:9200/\_plugin/marvel/kibana/index.html#/dashboard/file/marvel.overview.json](http://10.44.162.169:9200/_plugin/marvel/kibana/index.html#/dashboard/file/marvel.overview.json)
I have created an index via sense named myindex with a type named mytype to
push my data in it later.
I did also install kibana 4 and changed the kibana.yml like this :
# The host to bind the server to
host: "10.44.162.169"
# The Elasticsearch instance to use for all your queries.
elasticsearch\_url: "[http://10.44.162.169:9200](http://10.44.162.169:9200)"
I access to it via port 5601 (10.44.162.169:5601)
Now I want to load a data I have in my hdfs into my ElasticSearch.
After dowloading the es-hadoop jar and adding it to the path.
This is how I proceeded :
REGISTER /usr/elasticsearch-hadoop-2.0.2/dist/elasticsearch-hadoop-pig-2.0.2.jar
--load the CDR.csv file
cdr= LOAD '/user/omar/CDR.csv' using PigStorage(';')
AS TRAFFIC\_TYPE\_ID:int,APPELANT:int,CALLED\_NUMBER:int,CALL\_DURATION:int,LOCATION\_NUMBER:chararray,DATE\_HEURE\_APPEL:chararray);
STORE cdr INTO 'myindex/mytype' USING org.elasticsearch.hadoop.pig.PigRunner.run('es.nodes'='10.44.162.169');
When I execute this; the job is a success !!!
BUT, nothing seems to appear in my ES !
1. When I go and access to marvel, I don't find any documents in myindex !
2 )Neither in my Kibana plugin !
1. Furthermore, when I want to consult the logs in the HUE, I can't find a thing!
I then tried this way :
REGISTER /usr/elasticsearch-hadoop-2.0.2/dist/elasticsearch-hadoop-pig-2.0.2.jar
--load the CDR.csv file
cdr= LOAD '/user/admin/CDR\_OMAR.csv' using PigStorage(';')
AS
(traffic\_type\_id:int,caller:int,call\_time:datetime,tranche\_horaire:int,called:int,called:int,call\_duration:int,code\_type:chararray,code\_destination:chararray,location:chararray,id\_offre:int,id\_service:int,date\_heure\_appel:chararray);
--STORE cdr INTO 'indexOmar/typeOmar' USING EsStorage('es.nodes'='0.44.162.169:9200')
STORE cdr INTO 'telecom/cdr' USING org.elasticsearch.hadoop.pig.EsStorage('es.nodes'='10.44.162.169',
'es.mapping.names=call\_time:@timestamp',
'es.index.auto.create = false');
But, I got this error :
Run pig script using PigRunner.run() for Pig version 0.8+
2015-03-06 14:22:21,768 [main] INFO org.apache.pig.Main - Apache Pig version 0.12.0-cdh5.3.1 (rexported) compiled Jan 27 2015, 14:45:17
2015-03-06 14:22:21,770 [main] INFO org.apache.pig.Main - Logging error messages to: /yarn/nm/usercache/admin/appcache/application\_1425457357655\_0009/container\_1425457357655\_0009\_01\_000002/pig-job\_1425457357655\_0009.log
2015-03-06 14:22:21,863 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /var/lib/hadoop-yarn/.pigbootup not found
Any idea why this is happening and how to fix it?
Now the Hive issue :
I have downloaded the ES-Hadoop jar and added it to the path.
With that being said; I now want to load data from hive to ES.
1. First of all, I created a table via a CSV file under table metastore(with HUE)
2. I defined an external table on top of ES in hive to write and load data in it later:
ADD JAR
/usr/elasticsearch-hadoop-2.0.2/dist/elasticsearch-hadoop-hive-2.0.2.jar;
CREATE EXTERNAL TABLE es\_cdr(
id bigint,
calling int,
called int,
duration int,
location string,
date string)
ROW FORMAT SERDE 'org.elasticsearch.hadoop.hive.EsSerDe'
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES(
'es.nodes'='10.44.162.169',
'es.resource' = 'indexOmar/typeOmar');
I've also added manually the serde snapshot jar via paramaters=\> add file =\>jar
now, I want to load data from my table in the new ES table :
INSERT OVERWRITE TABLE es\_cdr
select NULL, h.appelant, h.called\_number, h.call\_duration, h.location\_number, h.date\_heure\_appel from hive\_cdr h;
but an error is appearing saying that :
Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
And this is what's written in the log :
15/03/05 14:36:34 INFO log.PerfLogger: \
15/03/05 14:36:34 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:\_col0, type:bigint, comment:null), FieldSchema(name:\_col1, type:int, comment:null), FieldSchema(name:\_col2, type:int, comment:null), FieldSchema(name:\_col3, type:int, comment:null), FieldSchema(name:\_col4, type:string, comment:null), FieldSchema(name:\_col5, type:string, comment:null)], properties:null)
15/03/05 14:36:34 INFO ql.Driver: EXPLAIN output for queryid hive\_20150305143636\_528f97d4-b670-40e2-ba80-7d7a7bd441ff : ABSTRACT SYNTAX TREE:
TOK\_QUERY
TOK\_FROM
TOK\_TABREF
TOK\_TABNAME
hive\_cdr
h
TOK\_INSERT
TOK\_DESTINATION
TOK\_TAB
TOK\_TABNAME
hive\_es\_cdr\_10
TOK\_SELECT
TOK\_SELEXPR
TOK\_NULL
TOK\_SELEXPR
.
TOK\_TABLE\_OR\_COL
h
appelant
TOK\_SELEXPR
.
TOK\_TABLE\_OR\_COL
h
called\_number
TOK\_SELEXPR
.
TOK\_TABLE\_OR\_COL
h
call\_dur
TOK\_SELEXPR
.
TOK\_TABLE\_OR\_COL
h
loc\_number
TOK\_SELEXPR
.
TOK\_TABLE\_OR\_COL
h
h\_appel
TOK\_LIMIT
2
STAGE DEPENDENCIES:
Stage-0 is a root stage [MAPRED]
STAGE PLANS:
Stage: Stage-0
Map Reduce
Map Operator Tree:
TableScan
alias: h
GatherStats: false
Select Operator
expressions: null (type: string), appelant (type: int), called\_number (type: int), call\_dur (type: int), loc\_number (type: string), h\_appel (type: string)
outputColumnNames: \_col0, \_col1, \_col2, \_col3, \_col4, \_col5
Limit
Number of rows: 2
Reduce Output Operator
sort order:
tag: -1
value expressions: \_col0 (type: void), \_col1 (type: int), \_col2 (type: int), \_col3 (type: int), \_col4 (type: string), \_col5 (type: string)
Path -\> Alias:
hdfs://master:8020/user/hive/warehouse/hive\_cdr [h]
Path -\> Partition:
hdfs://master:8020/user/hive/warehouse/hive\_cdr
Partition
base file name: hive\_cdr
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
properties:
COLUMN\_STATS\_ACCURATE true
bucket\_count -1
columns traffic\_type\_id,appelant,called\_number,call\_dur,loc\_number,h\_appel
columns.comments
columns.types int:int:int:int:string:string
field.delim ;
file.inputformat org.apache.hadoop.mapred.TextInputFormat
file.outputformat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
location hdfs://master:8020/user/hive/warehouse/hive\_cdr
name default.hive\_cdr
numFiles 1
numRows 0
rawDataSize 0
serialization.ddl struct hive\_cdr { i32 traffic\_type\_id, i32 appelant, i32 called\_number, i32 call\_dur, string loc\_number, string h\_appel}
serialization.format ;
serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
totalSize 56373362
transient\_lastDdlTime 1425459002
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
```
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
properties:
COLUMN_STATS_ACCURATE true
bucket_count -1
columns traffic_type_id,appelant,called_number,call_dur,loc_number,h_appel
columns.comments
columns.types int:int:int:int:string:string
field.delim ;
file.inputformat org.apache.hadoop.mapred.TextInputFormat
file.outputformat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
location hdfs://master:8020/user/hive/warehouse/hive_cdr
name default.hive_cdr
numFiles 1
numRows 0
rawDataSize 0
serialization.ddl struct hive_cdr { i32 traffic_type_id, i32 appelant, i32 called_number, i32 call_dur, string loc_number, string h_appel}
serialization.format ;
serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
totalSize 56373362
transient_lastDdlTime 1425459002
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
name: default.hive_cdr
name: default.hive_cdr
Truncated Path -> Alias:
/hive_cdr [h]
Needs Tagging: false
Reduce Operator Tree:
Extract
Limit
Number of rows: 2
Select Operator
expressions: UDFToLong(_col0) (type: bigint), _col1 (type: int), _col2 (type: int), _col3 (type: int), _col4 (type: string), _col5 (type: string)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
File Output Operator
compressed: false
GlobalTableId: 1
directory: hdfs://master:8020/user/hive/warehouse/hive_es_cdr_10
NumFilesPerFileSink: 1
Stats Publishing Key Prefix: hdfs://master:8020/user/hive/warehouse/hive_es_cdr_10/
table:
input format: org.elasticsearch.hadoop.hive.EsHiveInputFormat
jobProperties:
EXTERNAL TRUE
bucket_count -1
columns id_traffic,caller,called,call_dur,caller_location,call_date
columns.comments
columns.types bigint:int:int:int:string:string
es.nodes 10.44.162.169
es.port 9200
es.resource myindex/mytype
file.inputformat org.apache.hadoop.mapred.SequenceFileInputFormat
file.outputformat org.apache.hadoop.mapred.SequenceFileOutputFormat
location hdfs://master:8020/user/hive/warehouse/hive_es_cdr_10
name default.hive_es_cdr_10
serialization.ddl struct hive_es_cdr_10 { i64 id_traffic, i32 caller, i32 called, i32 call_dur, string caller_location, string call_date}
serialization.format 1
serialization.lib org.elasticsearch.hadoop.hive.EsSerDe
storage_handler org.elasticsearch.hadoop.hive.EsStorageHandler
transient_lastDdlTime 1425561441
output format: org.elasticsearch.hadoop.hive.EsHiveOutputFormat
properties:
EXTERNAL TRUE
bucket_count -1
columns id_traffic,caller,called,call_dur,caller_location,call_date
columns.comments
columns.types bigint:int:int:int:string:string
es.nodes 10.44.162.169
es.port 9200
es.resource myindex/mytype
file.inputformat org.apache.hadoop.mapred.SequenceFileInputFormat
file.outputformat org.apache.hadoop.mapred.SequenceFileOutputFormat
location hdfs://master:8020/user/hive/warehouse/hive_es_cdr_10
name default.hive_es_cdr_10
serialization.ddl struct hive_es_cdr_10 { i64 id_traffic, i32 caller, i32 called, i32 call_dur, string caller_location, string call_date}
serialization.format 1
serialization.lib org.elasticsearch.hadoop.hive.EsSerDe
storage_handler org.elasticsearch.hadoop.hive.EsStorageHandler
transient_lastDdlTime 1425561441
serde: org.elasticsearch.hadoop.hive.EsSerDe
name: default.hive_es_cdr_10
TotalFiles: 1
GatherStats: false
MultiFileSpray: false
```
15/03/05 14:36:34 INFO log.PerfLogger: \
15/03/05 14:36:34 INFO log.PerfLogger:
15/03/05 14:36:34 INFO log.PerfLogger:
15/03/05 14:36:34 INFO log.PerfLogger:
15/03/05 14:36:34 INFO lockmgr.DummyTxnManager: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
15/03/05 14:36:34 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=master:2181 sessionTimeout=600000 watcher=org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager$DummyWatcher@70e69669
15/03/05 14:36:34 INFO log.PerfLogger: \
15/03/05 14:36:34 INFO log.PerfLogger:
15/03/05 14:36:34 INFO ql.Driver: Starting command: INSERT OVERWRITE TABLE hive\_es\_cdr\_10
SELECT NULL,h.appelant,h.called\_number,h.call\_dur,h.loc\_number,h.h\_appel FROM hive\_cdr h limit 2
15/03/05 14:36:34 INFO ql.Driver: Total jobs = 1
15/03/05 14:36:34 INFO log.PerfLogger: \
15/03/05 14:36:34 INFO log.PerfLogger:
15/03/05 14:36:34 INFO log.PerfLogger:
15/03/05 14:36:34 INFO ql.Driver: Launching Job 1 out of 1
15/03/05 14:36:34 INFO exec.Task: Number of reduce tasks determined at compile time: 1
15/03/05 14:36:34 INFO exec.Task: In order to change the average load for a reducer (in bytes):
15/03/05 14:36:34 INFO exec.Task: set hive.exec.reducers.bytes.per.reducer=
15/03/05 14:36:34 INFO exec.Task: In order to limit the maximum number of reducers:
15/03/05 14:36:34 INFO exec.Task: set hive.exec.reducers.max=
15/03/05 14:36:34 INFO exec.Task: In order to set a constant number of reducers:
15/03/05 14:36:34 INFO exec.Task: set mapreduce.job.reduces=
15/03/05 14:36:34 INFO ql.Context: New scratch dir is hdfs://master:8020/tmp/hive-hive/hive\_2015-03-05\_14-36-34\_378\_4527939627221909415-7
15/03/05 14:36:34 INFO mr.ExecDriver: Using org.apache.hadoop.hive.ql.io.CombineHiveInputFormat
15/03/05 14:36:34 INFO mr.ExecDriver: adding libjars: file:///tmp/d39b23a8-98d2-4bc3-9008-3eff080dd20c\_resources/hive-serdes-1.0-SNAPSHOT.jar,file:///usr/elasticsearch-hadoop-2.0.2/dist/elasticsearch-hadoop-hive-2.0.2.jar,file:///opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hive/lib/hive-hbase-handler-0.13.1-cdh5.3.1.jar,file:///opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hbase/hbase-server.jar,file:///opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hbase/lib/htrace-core.jar,file:///opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hbase/lib/htrace-core-2.04.jar,file:///opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hbase/hbase-common.jar,file:///opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hbase/hbase-client.jar,file:///opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hbase/hbase-protocol.jar,file:///opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hbase/hbase-hadoop2-compat.jar,file:///opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hbase/hbase-hadoop-compat.jar
15/03/05 14:36:34 INFO exec.Utilities: Processing alias h
15/03/05 14:36:34 INFO exec.Utilities: Adding input file hdfs://master:8020/user/hive/warehouse/hive\_cdr
15/03/05 14:36:34 INFO exec.Utilities: Content Summary not cached for hdfs://master:8020/user/hive/warehouse/hive\_cdr
15/03/05 14:36:34 INFO ql.Context: New scratch dir is hdfs://master:8020/tmp/hive-hive/hive\_2015-03-05\_14-36-34\_378\_4527939627221909415-7
15/03/05 14:36:34 INFO log.PerfLogger:
15/03/05 14:36:34 INFO exec.Utilities: Serializing MapWork via kryo
15/03/05 14:36:34 INFO log.PerfLogger: \
15/03/05 14:36:34 INFO log.PerfLogger:
15/03/05 14:36:34 INFO exec.Utilities: Serializing ReduceWork via kryo
15/03/05 14:36:34 INFO log.PerfLogger: \
15/03/05 14:36:34 INFO client.RMProxy: Connecting to ResourceManager at master/10.44.162.169:8032
15/03/05 14:36:34 INFO client.RMProxy: Connecting to ResourceManager at master/10.44.162.169:8032
15/03/05 14:36:34 WARN mr.EsOutputFormat: Speculative execution enabled for reducer - consider disabling it to prevent data corruption
15/03/05 14:36:34 INFO mr.EsOutputFormat: Writing to [myindex/mytype]
15/03/05 14:36:34 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
15/03/05 14:36:35 INFO log.PerfLogger:
15/03/05 14:36:35 INFO io.CombineHiveInputFormat: CombineHiveInputSplit creating pool for hdfs://master:8020/user/hive/warehouse/hive\_cdr; using filter path hdfs://master:8020/user/hive/warehouse/hive\_cdr
15/03/05 14:36:35 INFO input.FileInputFormat: Total input paths to process : 1
15/03/05 14:36:35 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 3, size left: 0
15/03/05 14:36:35 INFO io.CombineHiveInputFormat: number of splits 1
15/03/05 14:36:35 INFO log.PerfLogger: \
15/03/05 14:36:35 INFO mapreduce.JobSubmitter: number of splits:1
15/03/05 14:36:36 INFO mapreduce.JobSubmitter: Submitting tokens for job: job\_1425457357655\_0006
15/03/05 14:36:36 INFO impl.YarnClientImpl: Submitted application application\_1425457357655\_0006
15/03/05 14:36:36 INFO mapreduce.Job: The url to track the job: [http://master:8088/proxy/application\_1425457357655\_0006/](http://master:8088/proxy/application_1425457357655_0006/)
15/03/05 14:36:36 INFO exec.Task: Starting Job = job\_1425457357655\_0006, Tracking URL = [http://master:8088/proxy/application\_1425457357655\_0006/](http://master:8088/proxy/application_1425457357655_0006/)
15/03/05 14:36:36 INFO exec.Task: Kill Command = /opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hadoop/bin/hadoop job -kill job\_1425457357655\_0006
15/03/05 14:36:58 INFO exec.Task: Hadoop job information for Stage-0: number of mappers: 0; number of reducers: 0
15/03/05 14:36:58 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
15/03/05 14:36:58 INFO exec.Task: 2015-03-05 14:36:58,687 Stage-0 map = 0%, reduce = 0%
15/03/05 14:36:58 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
15/03/05 14:36:58 ERROR exec.Task: Ended Job = job\_1425457357655\_0006 with errors
15/03/05 14:36:58 INFO impl.YarnClientImpl: Killed application application\_1425457357655\_0006
15/03/05 14:36:58 ERROR ql.Driver: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
15/03/05 14:36:58 INFO log.PerfLogger: \
15/03/05 14:36:58 INFO ql.Driver: MapReduce Jobs Launched:
15/03/05 14:36:58 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
15/03/05 14:36:58 INFO ql.Driver: Stage-Stage-0: HDFS Read: 0 HDFS Write: 0 FAIL
15/03/05 14:36:58 INFO ql.Driver: Total MapReduce CPU Time Spent: 0 msec
15/03/05 14:36:58 INFO log.PerfLogger:
15/03/05 14:36:58 INFO ZooKeeperHiveLockManager: about to release lock for default/hive\_es\_cdr\_10
15/03/05 14:36:58 INFO ZooKeeperHiveLockManager: about to release lock for default/hive\_cdr
15/03/05 14:36:58 INFO ZooKeeperHiveLockManager: about to release lock for default
15/03/05 14:36:58 INFO log.PerfLogger: \
15/03/05 14:36:58 ERROR operation.Operation: Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:147)
at org.apache.hive.service.cli.operation.SQLOperation.access$000(SQLOperation.java:69)
at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:502)
at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:213)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6315d306-d99e-4d66-8176-c610cc8868cb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6315d306-d99e-4d66-8176-c610cc8868cb%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@bdp](https://discuss.elastic.co/u/bdp)
#### Post date: [March 9, 2015, 4:16pm UTC](https://discuss.elastic.co/t/trouble-to-load-data-from-my-hadoop-cluster-to-elasticsearch-via-pig-and-hive/22580/2 "2015-03-09T16:16:04Z")
I am also trying to load data from hadoop to elasticsearch and also having some issues. I am facing different issue than yours.
In my case I have hadoop cluster and edgenode running on different machines than my elasticsearch server. First I confirmed that network communication is working between hadoop edge node and elasticsearch server. I have uploaded elasticsearch-hadoop-2.0.2.jar into my users directory on the edge node. And I also have a hive table present (shown below) that I want to upload into elasticsearch server running on a different machine.
hive\> show tables dw\_priority;
OK
tab\_name
Time taken: 0.064 seconds
hive\>
Then I added elasticsearch-hadoop-2.0.2.jar to the hive path as below:
hive\> ADD Jars ./elasticsearch-hadoop-2.0.2.jar;
Added ./elasticsearch-hadoop-2.0.2.jar to class path
Added resource: ./elasticsearch-hadoop-2.0.2.jar
hive\>
After that, when I try to create an external table into hive I get bunch of errors.
here is my external table definition followed by errors.
hive\> CREATE EXTERNAL TABLE remedytst\_es ( dw\_incident\_id double, incident\_number string)
\> ROW FORMAT SERDE 'org.elasticsearch.hadoop.hive.EsSerDe'
\> STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
\> TBLPROPERTIES (
\> 'es.nodes’=’64.102.212.139’ ,
\> 'es.resource’=‘itsm/incident’) ;
NoViableAltException(286@[])
at org.apache.hadoop.hive.ql.parse.HiveParser.tablePropertiesList(HiveParser.java:23403)
at org.apache.hadoop.hive.ql.parse.HiveParser.tableProperties(HiveParser.java:23292)
at org.apache.hadoop.hive.ql.parse.HiveParser.tablePropertiesPrefixed(HiveParser.java:23227)
at org.apache.hadoop.hive.ql.parse.HiveParser.createTableStatement(HiveParser.java:4460)
at org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:2016)
at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1298)
at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:938)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:190)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:342)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:977)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
FAILED: ParseException line 5:0 cannot recognize input near ''es.nodes’=’64.102.212.139’ ,\n'' 'es' '.' in table properties list
Looking at the errors, I think hive is not able to recognize the elasticsearch-hadoop-2.0.2.jar.
Did you also face the similar issue?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 12:27am UTC](https://discuss.elastic.co/t/trouble-to-load-data-from-my-hadoop-cluster-to-elasticsearch-via-pig-and-hive/22580/3 "2017-07-06T00:27:45Z")