# Unable to write to Elasticsearch from Spark Java

**URL:** <https://discuss.elastic.co/t/unable-to-write-to-elasticsearch-from-spark-java/231230>\
**Category:** Elasticsearch\
**Created:** [May 5, 2020, 10:18pm UTC](https://discuss.elastic.co/t/unable-to-write-to-elasticsearch-from-spark-java/231230 "2020-05-05T22:18:43Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![angryninja](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@angryninja](https://discuss.elastic.co/u/angryninja)\
**Post date:** [May 5, 2020, 10:18pm UTC](https://discuss.elastic.co/t/unable-to-write-to-elasticsearch-from-spark-java/231230/1 "2020-05-05T22:18:43Z")

</div>

Hi,

I have been following the documentation for writing data from Spark / Java into Elasticsearch mentioned here : [https://www.elastic.co/guide/en/elasticsearch/hadoop/current/spark.html#spark-sql](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/spark.html#spark-sql)  
But every time the documents are written it's just metadata and not the actual data from RDD.  
Is there any other config required to write to Elasticsearch from Spark/Java ?

I'm using ES v7.5.1 with spark v2.2.1 and elasticsearch-spark-20\_2.11 v7.5.1

Code :

```auto
 JavaSparkContext jsc = new JavaSparkContext(session.sparkContext());

// data to be saved
            Map<String, ?> otp = ImmutableMap.of("iata", "OTP", "name", "Otopeni");
            Map<String, ?> jfk = ImmutableMap.of("iata", "JFK", "name", "JFK NYC");

// create a pair RDD between the id and the docs
            JavaPairRDD<?, ?> pairRdd = jsc.parallelizePairs(ImmutableList.of(
                    new Tuple2<Object, Object>(1, otp),
                    new Tuple2<Object, Object>(2, jfk)));

            
JavaEsSpark.saveToEsWithMeta(pairRdd, "spark-index");

```

Documents created :

```auto
{
       "_index" : "spark-index",
       "_type" : "_doc",
       "_id" : "2",
       "_score" : 1.0
     },
     {
       "_index" : "spark-index",
       "_type" : "_doc",
       "_id" : "1",
       "_score" : 1.0
     }

```

---

<div class="post-metadata">

**Author:** ![Luca\_Belluccini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luca_belluccini/32/33239_2.png) [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)\
**Post date:** [May 6, 2020, 12:43am UTC](https://discuss.elastic.co/t/unable-to-write-to-elasticsearch-from-spark-java/231230/2 "2020-05-06T00:43:29Z")

</div>

Hello @angryninja,  
I think the problem might be related to [https://github.com/elastic/elasticsearch-hadoop/issues/913](https://github.com/elastic/elasticsearch-hadoop/issues/913)

Would it be possible to enable the logging on `org.elasticsearch.hadoop.rest` to `trace` and perform the same test? [Doc](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/logging.html)

---

<div class="post-metadata">

**Author:** ![angryninja](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@angryninja](https://discuss.elastic.co/u/angryninja)\
**Post date:** [May 6, 2020, 8:53pm UTC](https://discuss.elastic.co/t/unable-to-write-to-elasticsearch-from-spark-java/231230/3 "2020-05-06T20:53:45Z")

</div>

I looked at the issue but that is something else.  
I am now able to write to ES using following statement but saveToEs (both methods) don't work and I don't understand why. Both methods only write the metadata to ES and nothing else.

```auto
dataSetObj.write().format("org.elasticsearch.spark.sql").options(elasticSearchWriteOption()).mode("Append").save("spark-index");

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 3, 2020, 8:53pm UTC](https://discuss.elastic.co/t/unable-to-write-to-elasticsearch-from-spark-java/231230/4 "2020-06-03T20:53:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.