# Job fails injecting dataframe with variables in index name

**URL:** <https://discuss.elastic.co/t/job-fails-injecting-dataframe-with-variables-in-index-name/340370>\
**Category:** Elasticsearch\
**Created:** [August 8, 2023, 12:41pm UTC](https://discuss.elastic.co/t/job-fails-injecting-dataframe-with-variables-in-index-name/340370 "2023-08-08T12:41:29Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Joachim\_Rodrigues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joachim_rodrigues/32/97640_2.png) [@Joachim\_Rodrigues](https://discuss.elastic.co/u/Joachim_Rodrigues)\
**Post date:** [August 8, 2023, 12:41pm UTC](https://discuss.elastic.co/t/job-fails-injecting-dataframe-with-variables-in-index-name/340370/1 "2023-08-08T12:41:29Z")

</div>

Hello  
I have this code that injects a dataframe to an elastic cluster 7.9.3

`myDataframe.saveToEs("customer-{year}.{month}")`

But i'm getting this error :

```auto
User class threw exception: java.lang.Exception: Error(s) during treatment of stream data org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 4.0 failed 4 times, most recent failure: Lost task 0.3 in stage 4.0 (TID 1148, node06, executor 1): org.apache.spark.util.TaskCompletionListenerException: null
null
at org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:138)
at org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:116)
at org.apache.spark.scheduler.Task.run(Task.scala:137)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1405)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Driver stacktrace:
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:257)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:256)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

```

It starts injecting data but afert 2 mins it fails with this error,

But if i replace by :  
`myDataframe.saveToEs("customer-2023.8")`  
It wortks

And if i put again :  
`myDataframe.saveToEs("customer-{year}.{month}")`  
With superuser it works

But i can't let the superuser

Can any one help ?  
Thanks

---

<div class="post-metadata">

**Author:** ![Joachim\_Rodrigues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joachim_rodrigues/32/97640_2.png) [@Joachim\_Rodrigues](https://discuss.elastic.co/u/Joachim_Rodrigues)\
**Post date:** [August 10, 2023, 1:08pm UTC](https://discuss.elastic.co/t/job-fails-injecting-dataframe-with-variables-in-index-name/340370/2 "2023-08-10T13:08:57Z")

</div>

Found a fix  
The issue was that using **org.elasticsearch:elasticsearch-spark-20\_2.11:8.8.22** to send data  
But while looking at sent requests with wireshark there was a :  
**PUT /\_bulk** -\> ok it was my data sent and this request has been **ACK**

But just after there iwa a :  
**POST /\_refresh** with this status : **[FIN,ACK]**

This means that the client ended the connection

So i tried this **POST \_refresh** in postman and indeed this request returns **403 Forbidden**  
because this request was trying to access system indices : **.transform .security-07**

Finally i added this param to my spark session :

**.config(ES\_BATCH\_WRITE\_REFRESH, "false")**

And it worked, but i don't know if it was the correct way to fix the problem  
and also why

- suddenly it started to send this refresh
- or if suddenly the refresh was accessing those system indices ?

Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 7, 2023, 1:09pm UTC](https://discuss.elastic.co/t/job-fails-injecting-dataframe-with-variables-in-index-name/340370/3 "2023-09-07T13:09:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.