# Unable to index documents after around 20 million, Spark job fails because of NoNodesLeftException

**URL:** <https://discuss.elastic.co/t/unable-to-index-documents-after-around-20-million-spark-job-fails-because-of-nonodesleftexception/115518>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [January 15, 2018, 10:40am UTC](https://discuss.elastic.co/t/unable-to-index-documents-after-around-20-million-spark-job-fails-because-of-nonodesleftexception/115518 "2018-01-15T10:40:36Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ravi\_Ranjan](https://avatars.discourse-cdn.com/v4/letter/r/ecc23a/32.png) [@Ravi\_Ranjan](https://discuss.elastic.co/u/Ravi_Ranjan)\
**Post date:** [January 15, 2018, 10:40am UTC](https://discuss.elastic.co/t/unable-to-index-documents-after-around-20-million-spark-job-fails-because-of-nonodesleftexception/115518/1 "2018-01-15T10:40:36Z")

</div>

Hi,  
I am indexing some 30 million records in Elastic using pyspark. I ran the code and was able to index some 20 million records post which i got the following error:

```auto
18/01/15 10:34:42 WARN TaskSetManager: Lost task 100.0 in stage 2.0 (TID 102, localhost, executor driver): org.apache.spark.SparkException: Task failed while writing rows
	at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$.org$apache$spark$internal$io$SparkHadoopMapReduceWriter$$executeTask(SparkHadoopMapReduceWriter.scala:178)
	at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$$anonfun$3.apply(SparkHadoopMapReduceWriter.scala:89)
	at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$$anonfun$3.apply(SparkHadoopMapReduceWriter.scala:88)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:108)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[10.240.0.86:9200]] 
	at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:150)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:461)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:445)
	at org.elasticsearch.hadoop.rest.RestClient.bulk(RestClient.java:186)
	at org.elasticsearch.hadoop.rest.RestRepository.tryFlush(RestRepository.java:220)
	at org.elasticsearch.hadoop.rest.RestRepository.flush(RestRepository.java:242)
	at org.elasticsearch.hadoop.rest.RestRepository.doWriteToIndex(RestRepository.java:182)
	at org.elasticsearch.hadoop.rest.RestRepository.writeToIndex(RestRepository.java:159)
	at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.write(EsOutputFormat.java:151)
	at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$$anonfun$4.apply(SparkHadoopMapReduceWriter.scala:148)
	at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$$anonfun$4.apply(SparkHadoopMapReduceWriter.scala:144)
	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1375)
	at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$.org$apache$spark$internal$io$SparkHadoopMapReduceWriter$$executeTask(SparkHadoopMapReduceWriter.scala:159)
	... 8 more

```

How can I get rid of this error? I tried increasing the heap size, RAM and also changed host to external ip of the google server, to internal ip, to 0.0.0.0 and also localhost. Nothing seems to have solved this yet.

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [January 19, 2018, 6:05pm UTC](https://discuss.elastic.co/t/unable-to-index-documents-after-around-20-million-spark-job-fails-because-of-nonodesleftexception/115518/2 "2018-01-19T18:05:37Z")

</div>

If you increase the logging level on your job it should display the underlying communication failure information for the failed request. Another thing I'm noticing is that you only have a single node address being used. This isn't really a problem if you are using a cloud environment or single node deployment, but if you are using a multi-node deployment and have access to all nodes in the cluster, there may be an issue with your settings that is keeping the connector from discovering more nodes.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 16, 2018, 6:05pm UTC](https://discuss.elastic.co/t/unable-to-index-documents-after-around-20-million-spark-job-fails-because-of-nonodesleftexception/115518/3 "2018-02-16T18:05:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.