# Slow Performance of Elastic Search with Spark

**URL:** <https://discuss.elastic.co/t/slow-performance-of-elastic-search-with-spark/277101>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [June 25, 2021, 7:40pm UTC](https://discuss.elastic.co/t/slow-performance-of-elastic-search-with-spark/277101 "2021-06-25T19:40:11Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Hardik\_Aggarwal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hardik_aggarwal/32/85369_2.png) [@Hardik\_Aggarwal](https://discuss.elastic.co/u/Hardik_Aggarwal)\
**Post date:** [June 25, 2021, 7:40pm UTC](https://discuss.elastic.co/t/slow-performance-of-elastic-search-with-spark/277101/1 "2021-06-25T19:40:11Z")

</div>

I am trying to read the data from Elastic Search into a dataframe using Java ES-Spark-connector.. Now when I try to execute a query for example count() on the dataframe, the performance is dismal. For a 3Gb data it takes around 6 mins. On the other hand if I save the data in hadoop/hdfs it and then read it from there it takes around 3s. Can some one tell me a work around for this. The code I am using is as below.

```auto
SparkConf conf = new SparkConf().setAppName("Simple App").setMaster("local[*]");
        conf.set("es.index.auto.create", "true");
        JavaSparkContext javaSparkContext = new JavaSparkContext(conf);
       
        SparkSession spark = new SparkSession(javaSparkContext.sc());
        SQLContext sqlContext = new SQLContext(spark);
        Dataset<Row> ds = spark.read().format("org.elasticsearch.spark.sql").load("index_name");
       count = ds.count(); // this takes around 6 mins for 3GB data

```

---

<div class="post-metadata">

**Author:** ![priamai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priamai/32/80269_2.png) [@priamai](https://discuss.elastic.co/u/priamai)\
**Post date:** [June 29, 2021, 12:01am UTC](https://discuss.elastic.co/t/slow-performance-of-elastic-search-with-spark/277101/2 "2021-06-29T00:01:12Z")

</div>

I am also having performance problems, the same Spark query I do from the raw JSON files is 12-15 times faster than the same query via ES.  
I have about this is on the github issues but they told me to ask in here, anyway we need to find a way to inspect the push down query to understand what's going on.

---

<div class="post-metadata">

**Author:** ![priamai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priamai/32/80269_2.png) [@priamai](https://discuss.elastic.co/u/priamai)\
**Post date:** [July 1, 2021, 5:32am UTC](https://discuss.elastic.co/t/slow-performance-of-elastic-search-with-spark/277101/4 "2021-07-01T05:32:37Z")

</div>

And even if you do this:

`ds.count().explain(extended=True)`

does not show you the ES queries.

I have another discussion open here for monitoring [queries](https://discuss.elastic.co/t/monitoring-api-query/277200/2) in the backend which seems the only solution for now.

---

<div class="post-metadata">

**Author:** ![priamai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priamai/32/80269_2.png) [@priamai](https://discuss.elastic.co/u/priamai)\
**Post date:** [July 1, 2021, 2:48pm UTC](https://discuss.elastic.co/t/slow-performance-of-elastic-search-with-spark/277101/5 "2021-07-01T14:48:46Z")

</div>

Please [read this too](https://discuss.elastic.co/t/es-hadoop-push-down-aggregations-or-not/277514) because what is essentially happening is that the ES driver is not that smart and does just a full scroll of the data instead of leveraging the ES capabilities.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 29, 2021, 2:49pm UTC](https://discuss.elastic.co/t/slow-performance-of-elastic-search-with-spark/277101/6 "2021-07-29T14:49:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.