# Loading pySpark DataFrame into Elasticsearch - keep getting illegal argument error

**URL:** <https://discuss.elastic.co/t/loading-pyspark-dataframe-into-elasticsearch-keep-getting-illegal-argument-error/208348>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [November 18, 2019, 4:07pm UTC](https://discuss.elastic.co/t/loading-pyspark-dataframe-into-elasticsearch-keep-getting-illegal-argument-error/208348 "2019-11-18T16:07:43Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![dkajtoch](https://avatars.discourse-cdn.com/v4/letter/d/da6949/32.png) [@dkajtoch](https://discuss.elastic.co/u/dkajtoch)\
**Post date:** [November 18, 2019, 4:07pm UTC](https://discuss.elastic.co/t/loading-pyspark-dataframe-into-elasticsearch-keep-getting-illegal-argument-error/208348/1 "2019-11-18T16:07:43Z")

</div>

I am trying to send data stored in pySpark Dataframe directly into Elasticsearch. I found some code snippets online on how to do this and basically it looks like this:

```auto
# df - DataFrame
df.write.format("org.elasticsearch.spark.sql") \
   .option("es.nodes", "ip_address") \
   .option("es.resources", "test")
   .save()

```

but whatever I type in `option` results in IllegalArgumentException. I am running scala 2.11.0, spark 2.3.4 on Google Cloud. My elasticsearch machine is on a separate machine ver. 7.4.1  
I download jar `elasticsearch-spark-20_2.11-7.4.1.jar` and placed it in spark jars directory (it previously worked that way with mongo and google cloud storage).  
How to properly send pyspark DataFrame into Elasticsearch?

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [December 2, 2019, 6:04pm UTC](https://discuss.elastic.co/t/loading-pyspark-dataframe-into-elasticsearch-keep-getting-illegal-argument-error/208348/2 "2019-12-02T18:04:52Z")

</div>

This looks correct, could you share the IllegalArgumentException that you're getting here?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 30, 2019, 6:04pm UTC](https://discuss.elastic.co/t/loading-pyspark-dataframe-into-elasticsearch-keep-getting-illegal-argument-error/208348/3 "2019-12-30T18:04:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.