# Elasticsearch-hadoop-hive exception when writing array\\> column **URL:** \ **Category:** Elasticsearch\ **Created:** [March 12, 2015, 9:56pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-hive-exception-when-writing-array-map-string-string-column/22647 "2015-03-12T21:56:03Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Chen\_Wang](https://avatars.discourse-cdn.com/v4/letter/c/b5a626/32.png) [@Chen\_Wang](https://discuss.elastic.co/u/Chen_Wang)\ **Post date:** [March 12, 2015, 9:56pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-hive-exception-when-writing-array-map-string-string-column/22647/1 "2015-03-12T21:56:03Z")
Folks, I am using elasticsearch-hadoop-hive-2.1.0.Beta3.jar I defined the external table as:. CREATE EXTERNAL TABLE IF NOT EXISTS ${staging\_table}( customer\_id STRING, store\_purchase array\\>) ROW FORMAT SERDE 'org.elasticsearch.hadoop.hive.EsSerDe' STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' TBLPROPERTIES ( 'es.nodes'='localhost:9200', 'es.resource'='user\_activity/store', 'es.mapping.id'='customer\_id', 'es.input.json'='false', 'es.write.operation'='upsert', 'es.update.script'='ctx.\_source.store\_purchase += purchase', 'es.update.script.params'='purchase:store\_purchase' ) ;" I create another source table with the same column names and put some sample data. Running INSERT OVERWRITE TABLE ${staging\_table} ``` SELECT customer_id, store_purchase) FROM ${test_table} ``` but it throws EsHadoopIllegalArgumentException: Field [\_col1] needs to be a primitive; found [array\>]. Is array\> supported yet? If not, how can I get around this issue? Thanks~ -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/929c7b5b-fbb4-4232-821b-331499c18369%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/929c7b5b-fbb4-4232-821b-331499c18369%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [March 13, 2015, 4:18am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-hive-exception-when-writing-array-map-string-string-column/22647/2 "2015-03-13T04:18:08Z")
The exception occurs because you are trying to extract a field (the script parameters) from a complex type (array) and not a primitive. The issue with that (and why it's currently not supported) is because the internal structure of the complex type can get quite complex and its serialized, JSON form incorrect. Any reason why you need to pass the array of maps as a script parameter and not use primitives instead (you can use Hive column mapping to extract the ones you need)? On Thu, Mar 12, 2015 at 11:56 PM, Chen Wang [chen.apache.solr@gmail.com](mailto:chen.apache.solr@gmail.com) wrote: > Folks, > I am using elasticsearch-hadoop-hive-2.1.0.Beta3.jar > > I defined the external table as:. > CREATE EXTERNAL TABLE IF NOT EXISTS ${staging\_table}( > customer\_id STRING, > store\_purchase array\\>) > ROW FORMAT SERDE 'org.elasticsearch.hadoop.hive.EsSerDe' > STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' > TBLPROPERTIES ( > 'es.nodes'='localhost:9200', > 'es.resource'='user\_activity/store', > 'es.mapping.id'='customer\_id', > 'es.input.json'='false', > 'es.write.operation'='upsert', > 'es.update.script'='ctx.\_source.store\_purchase += purchase', > 'es.update.script.params'='purchase:store\_purchase' > ) ;" > > I create another source table with the same column names and put some > sample data. > > Running INSERT OVERWRITE TABLE ${staging\_table} > > ``` > SELECT customer_id, store_purchase) FROM ${test_table} > > ``` > > but it throws EsHadoopIllegalArgumentException: Field [\_col1] needs to be > a primitive; found [array\>]. Is array\> supported yet? If not, how can I get > around this issue? > > Thanks~ > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/929c7b5b-fbb4-4232-821b-331499c18369%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/929c7b5b-fbb4-4232-821b-331499c18369%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/929c7b5b-fbb4-4232-821b-331499c18369%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/929c7b5b-fbb4-4232-821b-331499c18369%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJogdmdtqpqyK%2BYGKDmkZwxFUJtpqty-q1Q1WrOAXwFB24zgaw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJogdmdtqpqyK%2BYGKDmkZwxFUJtpqty-q1Q1WrOAXwFB24zgaw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Chen\_Wang](https://avatars.discourse-cdn.com/v4/letter/c/b5a626/32.png) [@Chen\_Wang](https://discuss.elastic.co/u/Chen_Wang)\ **Post date:** [March 13, 2015, 4:54am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-hive-exception-when-writing-array-map-string-string-column/22647/3 "2015-03-13T04:54:00Z")
Costin, Thanks for your info. I am mapping array of maps to nested objects in ES, and in this specific case, the expected document in ES will look like { \_id:customer\_id, store\_purchase:[{item\_id:123, category:'pants', department:'clothes'}, ...] } so that I can do query like find all users between T1 and T2 ,have purchased items whose department is A and category is B. Anyway of achieving this with es-hadoop? Chen On Thursday, March 12, 2015 at 9:18:14 PM UTC-7, Costin Leau wrote: > The exception occurs because you are trying to extract a field (the script > parameters) from a complex type (array) and not a primitive. The issue with > that (and why it's currently not supported) is because the internal > structure of the complex type can get quite complex and its serialized, > JSON form incorrect. > Any reason why you need to pass the array of maps as a script parameter > and not use primitives instead (you can use Hive column mapping to extract > the ones you need)? > > On Thu, Mar 12, 2015 at 11:56 PM, Chen Wang \<[chen.apa...@gmail.com](mailto:chen.apa...@gmail.com) > \\> wrote: > > > Folks, > > I am using elasticsearch-hadoop-hive-2.1.0.Beta3.jar > > > > I defined the external table as:. > > CREATE EXTERNAL TABLE IF NOT EXISTS ${staging\_table}( > > customer\_id STRING, > > store\_purchase array\\>) > > ROW FORMAT SERDE 'org.elasticsearch.hadoop.hive.EsSerDe' > > STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' > > TBLPROPERTIES ( > > 'es.nodes'='localhost:9200', > > 'es.resource'='user\_activity/store', > > 'es.mapping.id'='customer\_id', > > 'es.input.json'='false', > > 'es.write.operation'='upsert', > > 'es.update.script'='ctx.\_source.store\_purchase += purchase', > > 'es.update.script.params'='purchase:store\_purchase' > > ) ;" > > > > I create another source table with the same column names and put some > > sample data. > > > > Running INSERT OVERWRITE TABLE ${staging\_table} > > > > ``` > > SELECT customer_id, store_purchase) FROM ${test_table} > > > > ``` > > > > but it throws EsHadoopIllegalArgumentException: Field [\_col1] needs to be > > a primitive; found [array\>]. Is array\> supported yet? If not, how can I get > > around this issue? > > > > Thanks~ > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/929c7b5b-fbb4-4232-821b-331499c18369%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/929c7b5b-fbb4-4232-821b-331499c18369%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/929c7b5b-fbb4-4232-821b-331499c18369%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/929c7b5b-fbb4-4232-821b-331499c18369%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/96c5b247-987b-45c5-bdfd-e9a690ad0215%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/96c5b247-987b-45c5-bdfd-e9a690ad0215%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:27am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-hive-exception-when-writing-array-map-string-string-column/22647/4 "2017-07-06T00:27:04Z")