I am trying to read the data from Elastic Search into a dataframe using Java ES-Spark-connector.. Now when I try to execute a query for example count() on the dataframe, the performance is dismal. For a 3Gb data it takes around 6 mins. On the other hand if I save the data in hadoop/hdfs it and then …

I am also having performance problems, the same Spark query I do from the raw JSON files is 12-15 times faster than the same query via ES. I have about this is on the github issues but they told me to ask in here, anyway we need to find a way to inspect the push down query to understand what's goin…

And even if you do this: ds.count().explain(extended=True) does not show you the ES queries. I have another discussion open here for monitoring queries in the backend which seems the only solution for now.

Slow Performance of Elastic Search with Spark

Elastic Stack Elasticsearch

priamai (Pangolin Security) July 1, 2021, 2:48pm 5

Please read this too because what is essentially happening is that the ES driver is not that smart and does just a full scroll of the data instead of leveraging the ES capabilities.

Topic		Replies	Views
Reading elasticsearch data using spark SQL is too slow Elasticsearch es-hadoop	1	715	April 3, 2020
Performance Challenge Elasticsearch es-hadoop	6	1141	April 28, 2017
Reading es by spark SQL is too slow Elasticsearch	1	401	March 18, 2020
Perofrmance problem on es-hadoop + spark Elasticsearch es-hadoop	5	1322	July 6, 2017
Elastic Search - JavaEsSpark read is slow Elasticsearch es-hadoop	8	1651	July 6, 2017

Slow Performance of Elastic Search with Spark

Related topics