# Spark Structured streaming

**URL:** <https://discuss.elastic.co/t/spark-structured-streaming/173871>\
**Category:** Elasticsearch\
**Created:** [March 26, 2019, 7:20am UTC](https://discuss.elastic.co/t/spark-structured-streaming/173871 "2019-03-26T07:20:59Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Fahad\_Khan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fahad_khan/32/42682_2.png) [@Fahad\_Khan](https://discuss.elastic.co/u/Fahad_Khan)\
**Post date:** [March 26, 2019, 7:20am UTC](https://discuss.elastic.co/t/spark-structured-streaming/173871/1 "2019-03-26T07:20:59Z")

</div>

HI there! I'm reading files from a folder via spark structured stream and I don't want to hard code the index name actually i want to utilize this.  
" Writing to dynamic/multi-resources[edit](https://github.com/elastic/elasticsearch-hadoop/edit/6.6/docs/src/reference/asciidoc/core/spark.adoc)

For cases when the data being written to Elasticsearch needs to be indexed under different buckets (based on the data content) one can use the `es.resource.write` field which accepts a pattern that is resolved from the document content, at runtime. Following the aforementioned [media example](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html#cfg-multi-writes), one could configure it as follows:

##### Scala[edit](https://github.com/elastic/elasticsearch-hadoop/edit/6.6/docs/src/reference/asciidoc/core/spark.adoc)

```
**val book = Map("media_type" -&gt; "book","title" -&gt; "Harry Potter","year" -&gt; "2010")**

```

\*\* val cd = Map("media\_type" -\> "music","title" -\> "Surfing With The Alien") \*\*  
\*\* val batch = sc.makeRDD(Seq(game, book, cd)) \*\*  
\*\* val microbatches = mutable.Queue(batch) \*\*  
\*\* ssc.queueStream(microbatches).saveToEs("my-collection-{media\_type}/doc") \*\*  
\*\* ssc.start()"\*\*  
and her is my code of line  
index= str(file\_path.split[-2])  
I have json in result variable and I'm appending in it a key value  
result['index']= index  
df.writeStream.option("checkpointLocation", "/home/user/chkpt\_stream6").outputMode("append").format("org.elasticsearch.spark.sql").option("checkpointLocation", "/home/user/spark\_chkpt9").option("es.resource","{index}/default").start()

But I'm getting an error  
org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot find match for {index}/default

any help would be really apprecaited

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 23, 2019, 7:21am UTC](https://discuss.elastic.co/t/spark-structured-streaming/173871/2 "2019-04-23T07:21:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.