# Unable to overwrite dataframe when documents have parent

**URL:** <https://discuss.elastic.co/t/unable-to-overwrite-dataframe-when-documents-have-parent/95283>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [August 1, 2017, 5:54am UTC](https://discuss.elastic.co/t/unable-to-overwrite-dataframe-when-documents-have-parent/95283 "2017-08-01T05:54:21Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Brendan\_Kerrison](https://avatars.discourse-cdn.com/v4/letter/b/53a042/32.png) [@Brendan\_Kerrison](https://discuss.elastic.co/u/Brendan_Kerrison)\
**Post date:** [August 1, 2017, 5:54am UTC](https://discuss.elastic.co/t/unable-to-overwrite-dataframe-when-documents-have-parent/95283/1 "2017-08-01T05:54:21Z")

</div>

I am trying to write a dataframe to Elasticsearch from Spark using ES-Hadoop 5.5.1. The documents are being written as children of pre-existing parent documents. Everything is working correctly the first time the DF is saved however it fails when attempting overwrite with:

```
  File "/usr/local/Cellar/apache-spark/2.1.1/libexec/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 550, in save
  File "/usr/local/Cellar/apache-spark/2.1.1/libexec/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
  File "/usr/local/Cellar/apache-spark/2.1.1/libexec/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
  File "/usr/local/Cellar/apache-spark/2.1.1/libexec/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o153.save.
: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: Found unrecoverable error [127.0.0.1:9205] returned Bad Request(400) - routing is required for [test_content]/[scorearticle]/[0-news-trending-movavg-scorearticle-ZW0382A001S00]; Bailing out..

```

Looking at the code, it does not look like the routing or parent of the documents being deleted is used when creating the bulk delete requests:

> <https://github.com/elastic/elasticsearch-hadoop/blob/master/mr/src/main/java/org/elasticsearch/hadoop/rest/RestRepository.java#L487>

Is there any way to get around this? Or do we need to look at deleting records ourselves prior to writing.

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [August 1, 2017, 9:18pm UTC](https://discuss.elastic.co/t/unable-to-overwrite-dataframe-when-documents-have-parent/95283/2 "2017-08-01T21:18:53Z")

</div>

Yeah, that would indeed be the case here. Thanks for bringing this to our attention. Could you open an issue on the Github project with this information?

---

<div class="post-metadata">

**Author:** ![Brendan\_Kerrison](https://avatars.discourse-cdn.com/v4/letter/b/53a042/32.png) [@Brendan\_Kerrison](https://discuss.elastic.co/u/Brendan_Kerrison)\
**Post date:** [August 1, 2017, 10:29pm UTC](https://discuss.elastic.co/t/unable-to-overwrite-dataframe-when-documents-have-parent/95283/3 "2017-08-01T22:29:40Z")

</div>

Will do.

---

<div class="post-metadata">

**Author:** ![Brendan\_Kerrison](https://avatars.discourse-cdn.com/v4/letter/b/53a042/32.png) [@Brendan\_Kerrison](https://discuss.elastic.co/u/Brendan_Kerrison)\
**Post date:** [August 1, 2017, 10:42pm UTC](https://discuss.elastic.co/t/unable-to-overwrite-dataframe-when-documents-have-parent/95283/4 "2017-08-01T22:42:40Z")

</div>

> <https://github.com/elastic/elasticsearch-hadoop/issues/1030>

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 29, 2017, 10:42pm UTC](https://discuss.elastic.co/t/unable-to-overwrite-dataframe-when-documents-have-parent/95283/5 "2017-08-29T22:42:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.