# Structured Streaming ES Sink

**URL:** <https://discuss.elastic.co/t/structured-streaming-es-sink/139501>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [July 11, 2018, 7:59am UTC](https://discuss.elastic.co/t/structured-streaming-es-sink/139501 "2018-07-11T07:59:10Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![mattjackson1989](https://avatars.discourse-cdn.com/v4/letter/m/ecb155/32.png) [@mattjackson1989](https://discuss.elastic.co/u/mattjackson1989)\
**Post date:** [July 11, 2018, 7:59am UTC](https://discuss.elastic.co/t/structured-streaming-es-sink/139501/1 "2018-07-11T07:59:10Z")

</div>

I have been trying to output data from my Spark Structured Streaming job to Elastic Search, following the example [here](https://discuss.elastic.co/t/spark-structured-streaming-sink-in-append-mode/105664/4).

I am using Spark 2.3.0 and elasticsearch-hadoop:6.3.1 (specified through --jars in spark submit).

The (python) code I am using (below) contains some additional configuration. Notice `.format("org.elasticsearch.spark.sql")` as opposed to `.format("es")`, following the advice [here](https://discuss.elastic.co/t/structured-streaming-failed-to-find-data-source-es/112144/3).

```auto
query = stream.writeStream \
              .outputMode("append") \
              .format("org.elasticsearch.spark.sql") \
              .option("es.nodes", <address>) \
              .option("es.port", "9200") \
              .option("es.nodes.discovery", "true") \
              .option("es.http.timeout", "20s") \
              .option("es.http.retries", "0") \
              .option("checkpointLocation", "~/checkpoint_es") \
              .start(indexTest/typeTest)

query.awaitTermination()

```

When I try to run this I am getting the following error.

```auto
org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot determine write shards for [structuredTest/testType]; likely its format is incorrect (maybe it contains illegal characters? or all shards failed?)

```

As far as I am aware this is not the case, but I can not get it working. Has anyone else seen this?

---

<div class="post-metadata">

**Author:** ![mattjackson1989](https://avatars.discourse-cdn.com/v4/letter/m/ecb155/32.png) [@mattjackson1989](https://discuss.elastic.co/u/mattjackson1989)\
**Post date:** [July 11, 2018, 8:40am UTC](https://discuss.elastic.co/t/structured-streaming-es-sink/139501/2 "2018-07-11T08:40:39Z")

</div>

**Update**

I realised that the upper case characters were causing the error.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 8, 2018, 8:54am UTC](https://discuss.elastic.co/t/structured-streaming-es-sink/139501/3 "2018-08-08T08:54:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.