# How to write to ES from a pyspark dataframe?

**URL:** <https://discuss.elastic.co/t/how-to-write-to-es-from-a-pyspark-dataframe/34728>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [November 16, 2015, 8:13pm UTC](https://discuss.elastic.co/t/how-to-write-to-es-from-a-pyspark-dataframe/34728 "2015-11-16T20:13:06Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![sendai](https://avatars.discourse-cdn.com/v4/letter/s/76d3ee/32.png) [@sendai](https://discuss.elastic.co/u/sendai)\
**Post date:** [November 16, 2015, 8:13pm UTC](https://discuss.elastic.co/t/how-to-write-to-es-from-a-pyspark-dataframe/34728/1 "2015-11-16T20:13:06Z")

</div>

I am not having any luck figuring out this. On the [docs](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/spark.html) there is code to read from ES not to write to ES. Can someone provide some documentation or examples?

---

<div class="post-metadata">

**Author:** ![eliasah](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eliasah/32/34741_2.png) [@eliasah](https://discuss.elastic.co/u/eliasah)\
**Post date:** [November 17, 2015, 8:30am UTC](https://discuss.elastic.co/t/how-to-write-to-es-from-a-pyspark-dataframe/34728/2 "2015-11-17T08:30:58Z")

</div>

Which version of Spark are you using?

---

<div class="post-metadata">

**Author:** ![sendai](https://avatars.discourse-cdn.com/v4/letter/s/76d3ee/32.png) [@sendai](https://discuss.elastic.co/u/sendai)\
**Post date:** [November 17, 2015, 12:44pm UTC](https://discuss.elastic.co/t/how-to-write-to-es-from-a-pyspark-dataframe/34728/3 "2015-11-17T12:44:16Z")

</div>

Thanks for the reply. I am using the spark-1.5.1-bin-hadoop2.6.tgz version downloaded from the spark site. The setup seems to work fine, I am just not sure how the code to write to ES would look like.

---

<div class="post-metadata">

**Author:** ![sendai](https://avatars.discourse-cdn.com/v4/letter/s/76d3ee/32.png) [@sendai](https://discuss.elastic.co/u/sendai)\
**Post date:** [November 20, 2015, 1:23pm UTC](https://discuss.elastic.co/t/how-to-write-to-es-from-a-pyspark-dataframe/34728/4 "2015-11-20T13:23:31Z")

</div>

What I found to write to ES from pyspark is to use saveAsNewAPIHadoopFile as showed below.

However this does not work for me with a dataframe.

```auto
> type(query)
pyspark.sql.dataframe.DataFrame
> conf = {"es.resource.write" : "logs/messages",
        "es.nodes" : "elasticsearch.domain.com:9200"
       }
> query.rdd.saveAsNewAPIHadoopFile(
    path='-', 
    outputFormatClass="org.elasticsearch.hadoop.mr.EsOutputFormat",
    keyClass="org.apache.hadoop.io.NullWritable", 
    valueClass="org.elasticsearch.hadoop.mr.LinkedMapWritable", 
    conf=conf)

```

I get this error:

```auto
Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.saveAsNewAPIHadoopFile.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 10.0 failed 4 times, most recent failure: Lost task 0.3 in stage 10.0 (TID 254, 10.244.10.114): net.razorvine.pickle.PickleException: expected zero arguments for construction of ClassDict (for pyspark.sql.types._create_row)
...

```

---

<div class="post-metadata">

**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\
**Post date:** [November 23, 2015, 9:51pm UTC](https://discuss.elastic.co/t/how-to-write-to-es-from-a-pyspark-dataframe/34728/5 "2015-11-23T21:51:59Z")

</div>

I'm afraid my pyspark experience is not vast enough to answer this question. It seems to be related to a class in Spark that cannot be properly created but I'm unclear way. Maybe the Spark list can provide more information...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:27pm UTC](https://discuss.elastic.co/t/how-to-write-to-es-from-a-pyspark-dataframe/34728/6 "2017-07-06T13:27:06Z")

</div>