# Spark / Elasticsearch Exception: Maybe ES was overloaded?

**URL:** <https://discuss.elastic.co/t/spark-elasticsearch-exception-maybe-es-was-overloaded/71932>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [January 18, 2017, 3:06am UTC](https://discuss.elastic.co/t/spark-elasticsearch-exception-maybe-es-was-overloaded/71932 "2017-01-18T03:06:55Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![rjurney](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rjurney/32/14662_2.png) [@rjurney](https://discuss.elastic.co/u/rjurney)\
**Post date:** [January 18, 2017, 3:06am UTC](https://discuss.elastic.co/t/spark-elasticsearch-exception-maybe-es-was-overloaded/71932/1 "2017-01-18T03:06:55Z")

</div>

I am having trouble testing some of the code from my new book, [Agile Data Science 2.0](http://shop.oreilly.com/product/0636920051619.do). I am writing from PySpark to Elasticsearch and keep running into an error.

This is local Spark on one node with local elasticsearch, as these are examples in a book. This is running on a r4.xlarge EC2 instance on Ubuntu. The Parquet data is 155MB.

The script is here: [ch04/pyspark\_to\_elasticsearch.py](https://github.com/rjurney/Agile_Data_Code_2/blob/master/ch04/pyspark_to_elasticsearch.py)

It looks like:

```
# Load the parquet file
on_time_dataframe = spark.read.parquet('data/on_time_performance.parquet')

on_time_dataframe.repartition(1).write.format("org.elasticsearch.spark.sql")\
  .repartition(1)\
  .option("es.resource","agile_data_science/on_time_performance")\
  .mode("overwrite")\
  .save()

```

Note that I added the call to repartition to try to throttle the work. After a few minutes, I get this error: [https://gist.github.com/rjurney/ec0d6b1ef050e3fbead2314255f4b6fa](https://gist.github.com/rjurney/ec0d6b1ef050e3fbead2314255f4b6fa)

The take home message is:

```
[agile_data_science][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest to [agile_data_science] containing [1000] requests]

```

What can I do to make this work? It looks like Elasticsearch is getting overloaded, but this is one Spark partition so I don't know how to throttle it further. Note that some records are getting written, I can search them afterwards but they don't all make it.

Would adding more shards help? I just don't know. Any suggestions would be appreciated.

Thanks!

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [February 3, 2017, 9:09pm UTC](https://discuss.elastic.co/t/spark-elasticsearch-exception-maybe-es-was-overloaded/71932/2 "2017-02-03T21:09:31Z")

</div>

@rjurney Are there any logs that appear in the Elasticsearch log that may highlight why the primary shard becomes inactive during the load?

---

<div class="post-metadata">

**Author:** ![rjurney](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rjurney/32/14662_2.png) [@rjurney](https://discuss.elastic.co/u/rjurney)\
**Post date:** [February 3, 2017, 11:39pm UTC](https://discuss.elastic.co/t/spark-elasticsearch-exception-maybe-es-was-overloaded/71932/3 "2017-02-03T23:39:47Z")

</div>

I fixed it by changing the batch size from 1,000 to 100.

```
    # Load the parquet file
    on_time_dataframe = spark.read.parquet('data/on_time_performance.parquet')

    # Save the DataFrame to Elasticsearch
    on_time_dataframe.write.format("org.elasticsearch.spark.sql")\
      .option("es.resource","agile_data_science/on_time_performance")\
      .option("es.batch.size.entries","100")\
      .mode("overwrite")\
      .save()
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 3, 2017, 11:39pm UTC](https://discuss.elastic.co/t/spark-elasticsearch-exception-maybe-es-was-overloaded/71932/4 "2017-03-03T23:39:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.