# \[Hadoop\] Slow performance of Elasticsearch-Hadoop + Spark SQL **URL:** \ **Category:** Elasticsearch\ **Created:** [June 1, 2015, 5:47pm UTC](https://discuss.elastic.co/t/hadoop-slow-performance-of-elasticsearch-hadoop-spark-sql/23738 "2015-06-01T17:47:41Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![dmitriyf](https://avatars.discourse-cdn.com/v4/letter/d/97f17d/32.png) [@dmitriyf](https://discuss.elastic.co/u/dmitriyf)\ **Post date:** [June 1, 2015, 5:47pm UTC](https://discuss.elastic.co/t/hadoop-slow-performance-of-elasticsearch-hadoop-spark-sql/23738/1 "2015-06-01T17:47:41Z")
Hi, I see a big difference in performance of the same query expressed via Spark SQL and CURL. In CURL the query runs less then a second, and in Spark SQL it runs 15 seconds. The index/type which I am querying contains 1M documents. Can you please explain why there is so big difference in performance? Are there any ways to tune performance of Elasticsearch + Spark SQL? Environment: (everything is running on the same box): Elasticsearch 1.4.4 elasticsearch-hadoop 2.1.0.BUILD-SNAPSHOT Spark 1.3.0. CURL: curl -XPOST "[http://localhost:9200/summary/intervals/\_search](http://localhost:9200/summary/intervals/_search)" -d' { "query" : { "filtered" : { "query" : { "match\_all" : {}}, "filter" : { "bool" : { "must" : [ { "term" : { "User" : "Robert Greene" } }, { "term" : { "DataStore" : "PROD\_HK\_HR" } }, { "term" : { "EventAffectedCount" : 56 } } ] } } } } }' Spark: ``` val sparkConf = new SparkConf().setAppName("Test1") // increasing scroll size to 5000 from the default 50 improved ``` performance by 2.5 times sparkConf.set("es.scroll.size", "5000") ``` val sc = new SparkContext(sparkConf) val sqlContext = new SQLContext(sc) val intv = sqlContext.esDF("summary/intervals") intv.registerTempTable("INTERVALS") val intv2 = sqlContext.sql("select EventCount, Hour " + "from intervals " + "where User = 'Robert Greene' " + "and DataStore = 'PROD_HK_HR' " + "and EventAffectedCount = 56 ") intv2.show(1000) ``` ## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/) You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8fc0d384-23bd-4807-8eae-a2ef2011f6ed%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8fc0d384-23bd-4807-8eae-a2ef2011f6ed%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [June 1, 2015, 9:40pm UTC](https://discuss.elastic.co/t/hadoop-slow-performance-of-elasticsearch-hadoop-spark-sql/23738/2 "2015-06-01T21:40:21Z")
The best way is to use a profiler to understand where time is spent. Spark while it is significantly faster than Hadoop, cannot compete with CULR. The latter is a simple REST connection - the former triggers a JVM, Scala, akka, Spark, which triggers es-hadoop which does the parallel call against all the nodes, retries the data in JSON format, converts it into Scala/Java and applies on schema on top for Spark SQL to run with. If you turn on logging, you'll see in fact there are multiple REST/CURL calls done by es-hadoop. With a JVM/Scala warmed up, you should see less than 15s however it depends on how much hardware you have available. Note that the curl comparison is not really fair - adding a SQL layer on top of that is bound to cost you something. On 6/1/15 8:47 PM, Dmitriy Fingerman wrote: > Hi, > > I see a big difference in performance of the same query expressed via Spark SQL and CURL. > In CURL the query runs less then a second, and in Spark SQL it runs 15 seconds. > The index/type which I am querying contains 1M documents. > Can you please explain why there is so big difference in performance? > Are there any ways to tune performance of Elasticsearch + Spark SQL? > > Environment: (everything is running on the same box): > Elasticsearch 1.4.4 > elasticsearch-hadoop 2.1.0.BUILD-SNAPSHOT > Spark 1.3.0. > > CURL: > > curl -XPOST "[http://localhost:9200/summary/intervals/\_search](http://localhost:9200/summary/intervals/_search)" -d' > { > "query" : { > "filtered" : { > "query" : { "match\_all" : {}}, > "filter" : { > "bool" : { > "must" : [ > { > "term" : { "User" : "Robert Greene" } > }, > { > "term" : { "DataStore" : "PROD\_HK\_HR" } > }, > { > "term" : { "EventAffectedCount" : 56 } > } > ] > } > } > } > } > }' > > Spark: > > ``` > val sparkConf = new SparkConf().setAppName("Test1") > > // increasing scroll size to 5000 from the default 50 improved performance by 2.5 times > sparkConf.set("es.scroll.size", "5000") > > val sc = new SparkContext(sparkConf) > val sqlContext = new SQLContext(sc) > > val intv = sqlContext.esDF("summary/intervals") > intv.registerTempTable("INTERVALS") > > val intv2 = sqlContext.sql("select EventCount, Hour " + > "from intervals " + > "where User = 'Robert Greene' " + > "and DataStore = 'PROD_HK_HR' " + > "and EventAffectedCount = 56 ") > intv2.show(1000) > > ``` > > ## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/) > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/8fc0d384-23bd-4807-8eae-a2ef2011f6ed%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8fc0d384-23bd-4807-8eae-a2ef2011f6ed%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/8fc0d384-23bd-4807-8eae-a2ef2011f6ed%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8fc0d384-23bd-4807-8eae-a2ef2011f6ed%40googlegroups.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Costin ## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/) You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/556CD145.3010203%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/556CD145.3010203%40gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:10am UTC](https://discuss.elastic.co/t/hadoop-slow-performance-of-elasticsearch-hadoop-spark-sql/23738/3 "2017-07-06T00:10:19Z")