# Allow multivalued/array for all fields?

**URL:** https://discuss.elastic.co/t/allow-multivalued-array-for-all-fields/107060
**Category:** Elasticsearch
**Tags:** es-hadoop
**Created:** [November 9, 2017, 3:47pm UTC](https://discuss.elastic.co/t/allow-multivalued-array-for-all-fields/107060 "2017-11-09T15:47:37Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![ghukill](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ghukill/32/24032_2.png) [@ghukill](https://discuss.elastic.co/u/ghukill)
#### Post date: [November 9, 2017, 3:47pm UTC](https://discuss.elastic.co/t/allow-multivalued-array-for-all-fields/107060/1 "2017-11-09T15:47:37Z")

</div>

I'm attempting to write a Hadoop RDD to ElasticSearch, without the luxury of seeing the fields before they are created. So far, dynamic mapping has been working well when the values for fields are singular. But, when I attempt to write fields with a list/array, I'm getting errors.

I found [this post](https://discuss.elastic.co/t/pyspark-write-list-type-to-es/43944) which I hoped would solve my problem, but no luck yet.

I'm also a bit new to ElasticSearch still. I'm wondering, is it possible to use the `es.read.field.as.array.include` and apply it to all fields? Something like:

```
conf={
	"es.resource":"j34/record",
	"es.nodes":"192.168.45.10:9200",
	"es.mapping.exclude":"temp_id",
	"es.mapping.id":"temp_id",
	"es.read.field.as.array.include":"*" # this setting specifically
	}

```

With the understanding that all fields would then be arrays? Thanks for any suggestions or advice.

As I type this, wondering if I would need to exclude fields like `temp_id` and `_id`?

The specific error I'm getting:  
`org.apache.spark.SparkException: Data of type java.util.ArrayList cannot be used at org.apache.spark.api.python.JavaToWritableConverter.org$apache$spark$api$python$JavaToWritableConverter$$convertToWritable(PythonHadoopUtil.scala:141) at org.apache.spark.api.python.JavaToWritableConverter$$anonfun$org$apache$spark$api$python$JavaToWritableConverter$$convertToWritable$1.apply(PythonHadoopUtil.scala:134)`

---

<div class="post-metadata">

### Author: ![ghukill](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ghukill/32/24032_2.png) [@ghukill](https://discuss.elastic.co/u/ghukill)
#### Post date: [November 9, 2017, 4:06pm UTC](https://discuss.elastic.co/t/allow-multivalued-array-for-all-fields/107060/2 "2017-11-09T16:06:59Z")

</div>

My first post was short-lived, thanks to [this StackOverflow post](https://stackoverflow.com/a/37393087/1196358).

The solution was not to use `es.read.field.as.array.include`, but convert all python lists to tuples before handing over to RDD, and eventually `saveAsNewAPIHadoopFile()`.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 7, 2017, 4:07pm UTC](https://discuss.elastic.co/t/allow-multivalued-array-for-all-fields/107060/3 "2017-12-07T16:07:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
