# Elasticsearch es-spark too many request **URL:** \ **Category:** Elasticsearch\ **Created:** [September 16, 2019, 2:58pm UTC](https://discuss.elastic.co/t/elasticsearch-es-spark-too-many-request/199684 "2019-09-16T14:58:50Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Vishnu\_Singhal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishnu_singhal/32/52488_2.png) [@Vishnu\_Singhal](https://discuss.elastic.co/u/Vishnu_Singhal)\ **Post date:** [September 16, 2019, 2:58pm UTC](https://discuss.elastic.co/t/elasticsearch-es-spark-too-many-request/199684/1 "2019-09-16T14:58:50Z")
I'm getting continuously org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 109896.0 failed 4 times, most recent failure: Lost task 0.3 in stage 109896.0 (TID 248952, 10.244.8.28, executor 2): org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: [HEAD] on [abc\_destination\_owasp] failed; server[10.99.57.250:9200] returned [429|Too Many Requests:] at org.elasticsearch.hadoop.rest.RestClient.checkResponse(RestClient.java:469) at org.elasticsearch.hadoop.rest.RestClient.executeNotFoundAllowed(RestClient.java:439) at org.elasticsearch.hadoop.rest.RestClient.exists(RestClient.java:529) at org.elasticsearch.hadoop.rest.RestClient.indexExists(RestClient.java:524) at org.elasticsearch.hadoop.rest.RestClient.touch(RestClient.java:535) at org.elasticsearch.hadoop.rest.RestRepository.touch(RestRepository.java:364) at org.elasticsearch.hadoop.rest.RestService.initSingleIndex(RestService.java:660) at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:636) at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:65) at org.elasticsearch.spark.rdd.EsSpark$$anonfun$doSaveToEs$1.apply(EsSpark.scala:108) at org.elasticsearch.spark.rdd.EsSpark$$anonfun$doSaveToEs$1.apply(EsSpark.scala:108) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:121) at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1887) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1875) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1874) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1874) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926) at scala.Option.foreach(Option.scala:257) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2108) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2057) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2046) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2114) at org.elasticsearch.spark.rdd.EsSpark$.doSaveToEs(EsSpark.scala:108) at org.elasticsearch.spark.rdd.EsSpark$.saveToEs(EsSpark.scala:79) at org.elasticsearch.spark.rdd.EsSpark$.saveToEs(EsSpark.scala:76) at org.elasticsearch.spark.rdd.EsSpark$.saveJsonToEs(EsSpark.scala:114) at org.elasticsearch.spark.rdd.api.java.JavaEsSpark$.saveJsonToEs(JavaEsSpark.scala:63) at org.elasticsearch.spark.rdd.api.java.JavaEsSpark.saveJsonToEs(JavaEsSpark.scala) at com.abc.def.pipeline.processor.ABC.lambda$null$46890760$1(UserEventDRProcessor.java:83) at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$1.apply(JavaDStreamLike.scala:272) at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$1.apply(JavaDStreamLike.scala:272) at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:628) at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:628) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51) at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50) at scala.util.Try$.apply(Try.scala:192) at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39) at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:257) at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257) at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:256) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) I'm using this version of jar compile group: 'org.elasticsearch', name: 'elasticsearch-spark-20\_2.11', version: '7.1.1' ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 16, 2019, 4:26pm UTC](https://discuss.elastic.co/t/elasticsearch-es-spark-too-many-request/199684/2 "2019-09-16T16:26:29Z")
What is the size of your Elasticsearch cluster? How many indices and shards are you actively indexing into? How many clients do you have indexing into the cluster? ---
**Author:** ![Vishnu\_Singhal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishnu_singhal/32/52488_2.png) [@Vishnu\_Singhal](https://discuss.elastic.co/u/Vishnu_Singhal)\ **Post date:** [September 18, 2019, 4:57am UTC](https://discuss.elastic.co/t/elasticsearch-es-spark-too-many-request/199684/3 "2019-09-18T04:57:02Z")
> [@Christian\_Dahlqvist](#): > > What is the size of your Elasticsearch cluster? How many indices and shards are you actively indexing into? How many clients do you have indexing into the cluster? Size of Cluster is 3 indices count is 6 I'm using all default configuration provided in this helm chart > **[GitHub - elastic/elasticsearch: Free and Open, Distributed, RESTful Search...](https://github.com/elastic/elasticsearch)** > > Free and Open, Distributed, RESTful Search Engine. Contribute to elastic/elasticsearch development by creating an account on GitHub. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 18, 2019, 5:17am UTC](https://discuss.elastic.co/t/elasticsearch-es-spark-too-many-request/199684/4 "2019-09-18T05:17:29Z")
The error indicated that you are overwhelming the cluster. I would recommend you read [this blog post](https://www.elastic.co/blog/why-am-i-seeing-bulk-rejections-in-my-elasticsearch-cluster) for further details. How much RAM and CPU does each node have allocated? Are you using locally attached SSDs for storage? How many processes/threads are concurrently writing to the cluster? ---
**Author:** ![Vishnu\_Singhal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishnu_singhal/32/52488_2.png) [@Vishnu\_Singhal](https://discuss.elastic.co/u/Vishnu_Singhal)\ **Post date:** [September 20, 2019, 2:52pm UTC](https://discuss.elastic.co/t/elasticsearch-es-spark-too-many-request/199684/5 "2019-09-20T14:52:02Z")
yes. its local ssd on the k8s cluster. Its not more then 2-3 threads that writing on to this cluster. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 18, 2019, 2:52pm UTC](https://discuss.elastic.co/t/elasticsearch-es-spark-too-many-request/199684/6 "2019-10-18T14:52:06Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.