# Inserting data from Spark application creates unassigned shards

**URL:** <https://discuss.elastic.co/t/inserting-data-from-spark-application-creates-unassigned-shards/56257>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [July 25, 2016, 3:22am UTC](https://discuss.elastic.co/t/inserting-data-from-spark-application-creates-unassigned-shards/56257 "2016-07-25T03:22:30Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Zoran\_Jeremic](https://avatars.discourse-cdn.com/v4/letter/z/7ba0ec/32.png) [@Zoran\_Jeremic](https://discuss.elastic.co/u/Zoran_Jeremic)\
**Post date:** [July 25, 2016, 3:22am UTC](https://discuss.elastic.co/t/inserting-data-from-spark-application-creates-unassigned-shards/56257/1 "2016-07-25T03:22:30Z")

</div>

I have a scala spark application that should store some data to ES 2.3.3. I'm trying to insert some very simple data using the following code:

> val sparkConf = new SparkConf()  
> .setMaster(master)  
> .set("spark.cores\_max",maxCores.toString)  
> .set("spark.executor.memory","4g")  
> .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")  
> .setAppName("testapp")  
> .set("es.index.auto.create","true")  
> .set("es.nodes", "127.0.0.1")  
> .set("es.port","9200")  
> .set("es.http.timeout","5m")  
> .set("es.scroll.size","50")  
> val sc = new SparkContext(sparkConf)

> val r2 = Map("arrival" -\> "LA", "SFO" -\> "San Fran")  
> val rdd = sc.makeRDD(Seq(r2))  
> EsSpark.saveToEs(rdd, "spark/docs")

When this code is executed it inserts data, but have unassigned shards and cluster state is Yellow. I tried to check from logs, but didn't find something use. It seems that 4 shards are created and then the following error happens before fifth shard is created:

> [2016-07-24 20:11:09,369][DEBUG][cluster.service] [Cottonmouth] cluster state update task [create-index [spark], cause [api]] failed  
> [spark] IndexAlreadyExistsException[already exists]  
> at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.validateIndexName(MetaDataCreateIndexService.java:138)  
> at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.validate(MetaDataCreateIndexService.java:473)  
> at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.access$100(MetaDataCreateIndexService.java:97)  
> at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.execute(MetaDataCreateIndexService.java:192)  
> at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45)  
> at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:468)  
> at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)  
> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)  
> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)  
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  
> at java.lang.Thread.run(Thread.java:745)  
> [2016-07-24 20:11:09,372][DEBUG][cluster.service] [Cottonmouth] processing [create-index [spark], cause [api]]: took 3ms no change in cluster\_state  
> [2016-07-24 20:11:09,372][DEBUG][rest.suppressed] /spark Params: {index=spark}  
> [spark] IndexAlreadyExistsException[already exists]  
> at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.validateIndexName(MetaDataCreateIndexService.java:138)  
> at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.validate(MetaDataCreateIndexService.java:473)  
> at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.access$100(MetaDataCreateIndexService.java:97)  
> at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.execute(MetaDataCreateIndexService.java:192)  
> at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45)  
> at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:468)  
> at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)  
> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)  
> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)  
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  
> at java.lang.Thread.run(Thread.java:745)  
> [2016-07-24 20:11:09,410][DEBUG][index.engine] [Cottonmouth] [spark][0] no translog ID present in the current generation - creating one

Any idea how to fix the problem?

---

<div class="post-metadata">

**Author:** ![jkuang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jkuang/32/72637_2.png) [@jkuang](https://discuss.elastic.co/u/jkuang)\
**Post date:** [April 12, 2017, 10:05pm UTC](https://discuss.elastic.co/t/inserting-data-from-spark-application-creates-unassigned-shards/56257/2 "2017-04-12T22:05:24Z")

</div>

The error message said Index already exist, but you're saying it still creates the index and then some unassigned shards?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:22pm UTC](https://discuss.elastic.co/t/inserting-data-from-spark-application-creates-unassigned-shards/56257/3 "2017-07-06T13:22:10Z")

</div>