Elasticsearch hadoop - java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.Text

Elastic Stack Elasticsearch
1

Hi,

I'm using elasticsearch 1.0 GA and elasticsearch-hadoop-1.3.0.M2.jar with
hive 0.11

I created external table like

CREATE EXTERNAL TABLE estable (
data STRUCT<
intmap:Map<STRING, INT>
>
)

and when i try to insert data into this table, i get this exception

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.
ClassCastException: org.apache.hadoop.io.IntWritable cannot be cast to org.
apache.hadoop.io.Text
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(
GroupByOperator.java:829)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502
)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832
)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(
SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502
)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832
)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(
GroupByOperator.java:1066)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(
GroupByOperator.java:890)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(
GroupByOperator.java:758)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(
GroupByOperator.java:824)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502
)
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.
java:253)
... 3 more
Caused by: java.lang.ClassCastException: org.apache.hadoop.io.IntWritablecannot be cast to org
.apache.hadoop.io.Text
at org.apache.hadoop.hive.serde2.objectinspector.primitive.
WritableStringObjectInspector.copyObject(WritableStringObjectInspector.java:
35)
at org.apache.hadoop.hive.serde2.objectinspector.
ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:249)
at org.apache.hadoop.hive.serde2.objectinspector.
ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:229)
at brickhouse.udf.collect.CollectUDAF$MapCollectUDAFEvaluator.
putIntoSet(CollectUDAF.java:252)
at brickhouse.udf.collect.CollectUDAF$MapCollectUDAFEvaluator.
iterate(CollectUDAF.java:222)
at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.
aggregate(GenericUDAFEvaluator.java:159)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations
(GroupByOperator.java:663)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(
GroupByOperator.java:916)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(
GroupByOperator.java:758)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(
GroupByOperator.java:824)
... 14 more

If i create a external table with Map<STRING, STRING>, like

CREATE EXTERNAL TABLE estable (
data STRUCT<
intmap:Map<STRING, STRING>
>
)

everything works fine

Is it bug? or did i do something wrong? How can i fix it?

Thanks for helps.

--
Best
moon

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/835efed7-8f36-4025-b17e-bcab3a22047c%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

You're Hive mapping needs to match the Elasticsearch one - there's no type conversion applied between the two.

The ClassCastException indicates that you are trying to write an INT (IntWritable) to a STRING (Text) - hence when you
change your mapping to STRING, STRING things are working.
I assume you already have some data in Elasticsearch and you are trying to read it in Hive through es-hadoop. However
your ES mapping points to a String not an Integer hence the type issue.

Either fix your mapping in ES or load the data as a String and then in Hive do the type conversion (casting) to int.

On 2/23/2014 2:27 AM, moon soo Lee wrote:

Hi,

I'm using elasticsearch 1.0 GA and elasticsearch-hadoop-1.3.0.M2.jar with hive 0.11

I created external table like

|
CREATE EXTERNAL TABLE estable (
data STRUCT<
intmap:Map<STRING,INT>

)
|

and when i try to insert data into this table, i get this exception

|
Causedby:org.apache.hadoop.hive.ql.metadata.HiveException:java.lang.ClassCastException:org.apache.hadoop.io.IntWritablecannot
be cast to org.apache.hadoop.io.Text
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:829)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1066)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(GroupByOperator.java:890)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:758)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:824)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:253)
...3more
Causedby:java.lang.ClassCastException:org.apache.hadoop.io.IntWritablecannot be cast to org.apache.hadoop.io.Text
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.copyObject(WritableStringObjectInspector.java:35)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:249)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:229)
at brickhouse.udf.collect.CollectUDAF$MapCollectUDAFEvaluator.putIntoSet(CollectUDAF.java:252)
at brickhouse.udf.collect.CollectUDAF$MapCollectUDAFEvaluator.iterate(CollectUDAF.java:222)
at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:159)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:663)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(GroupByOperator.java:916)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:758)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:824)
...14more
|

If i create a external table with Map<STRING, STRING>, like

|
CREATE EXTERNAL TABLE estable (
data STRUCT<
intmap:Map<STRING,STRING>

)
|

everything works fine

Is it bug? or did i do something wrong? How can i fix it?

Thanks for helps.

--
Best
moon

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/835efed7-8f36-4025-b17e-bcab3a22047c%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/530A0D96.10708%40gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.