# Save PySpark DataFRame to ElasticSEarch index

**URL:** <https://discuss.elastic.co/t/save-pyspark-dataframe-to-elasticsearch-index/244581>\
**Category:** Elasticsearch\
**Created:** [August 11, 2020, 3:13pm UTC](https://discuss.elastic.co/t/save-pyspark-dataframe-to-elasticsearch-index/244581 "2020-08-11T15:13:23Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![larryrodrigues](https://avatars.discourse-cdn.com/v4/letter/l/94ad74/32.png) [@larryrodrigues](https://discuss.elastic.co/u/larryrodrigues)\
**Post date:** [August 11, 2020, 3:13pm UTC](https://discuss.elastic.co/t/save-pyspark-dataframe-to-elasticsearch-index/244581/1 "2020-08-11T15:13:23Z")

</div>

Using Spark 2.6 and elasticsearch 6.8

I am trying to save a PySpark DataFrame to elasticsearch index using the external library:

df\_to\_save.write.format('org.elasticsearch.spark.sql').option('es.resource', '%s/%s' % ('spark\_obj\_2020', '\_doc'),).save()

The index I am saving to has a template definition with complex/object types. The DataFrame saves to the index, but all data is saved as Strings. The JSON Structure is lost, and a column converted to JSON type Struct gets converted to string data.

Index template looks like:

```auto
  "mappings": {
    "_doc": {
        "dynamic":"strict",
        "properties": {
          "host":{
            "properties": {
              "address": {
                "type":"keyword",
                "fields": {
                  "text": {
                    "type":"text"
                  }
                }
              },
              "ip": {"type":"ip"},
              "jam": {
                 "properties": {
                   "ink": {
                     "type": "keyword",
                     "fields": {
                       "text": {
                         "type":"text"
                       }
                     }
                   },
                   "stream": {
                     "type": "keyword",
                     "fields": {
                       "text": {
                         "type":"text"
                       }
                     }
                   }
                 }
              }
            }
          }

```

Spark schema:  
root  
|-- host: struct (nullable = false)  
| |-- host: string (nullable = true)  
|-- method: string (nullable = true)  
|-- date: string (nullable = true)

The host data contains a JSON String with matching structure.

---

<div class="post-metadata">

**Author:** ![larryrodrigues](https://avatars.discourse-cdn.com/v4/letter/l/94ad74/32.png) [@larryrodrigues](https://discuss.elastic.co/u/larryrodrigues)\
**Post date:** [August 12, 2020, 5:08pm UTC](https://discuss.elastic.co/t/save-pyspark-dataframe-to-elasticsearch-index/244581/2 "2020-08-12T17:08:15Z")

</div>

**Found the fix:**

1. Fix DataFrame column create and Type, leave as Struct do not convert to JSON:  
#df\_to\_save = df\_to\_save.withColumn("host", F.to\_json(F.struct("address","ip","jam")) )  
df\_to\_save = df\_to\_save.withColumn("host", F.struct("address","ip","jam"))

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 9, 2020, 5:08pm UTC](https://discuss.elastic.co/t/save-pyspark-dataframe-to-elasticsearch-index/244581/3 "2020-09-09T17:08:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.