# Reading from Elasticsearch index using spark ( es-hadoop ) connectors **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [February 18, 2022, 10:22am UTC](https://discuss.elastic.co/t/reading-from-elasticsearch-index-using-spark-es-hadoop-connectors/297590 "2022-02-18T10:22:35Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Khushboo\_Kaul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/khushboo_kaul/32/100884_2.png) [@Khushboo\_Kaul](https://discuss.elastic.co/u/Khushboo_Kaul)\ **Post date:** [February 18, 2022, 10:22am UTC](https://discuss.elastic.co/t/reading-from-elasticsearch-index-using-spark-es-hadoop-connectors/297590/1 "2022-02-18T10:22:35Z")
HI Team, I was trying to read the index from Elasticsearch using es-hadoop connectors using spark using the below code- The index contains millions of records and the read is super slow. is there anything I can do to do better the read? es\_df = spark.read.format("org.Elasticsearch.spark.sql") .option("es.nodes.wan.only","true") .option("es.port",port) .option("es.net.ssl","false") .option("es.nodes", "\*\*\*") .load(f"{final\_resource}") .select("col1","col2") ---
**Author:** ![Keith\_Massey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keith_massey/32/83666_2.png) [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)\ **Post date:** [February 22, 2022, 3:04pm UTC](https://discuss.elastic.co/t/reading-from-elasticsearch-index-using-spark-es-hadoop-connectors/297590/2 "2022-02-22T15:04:39Z")
Hi @Khushboo_Kaul. The answer is almost certainly "yes". I'd probably start with checking the number of executors you are configured to use in spark. You want as much parallelism as your cluster can handle. You might have to experiment a little to find out what the best number is. You want at least one per Elasticsearch data node (and you can probably handle a good bit more than that). Another thing I'd check is the `es.scroll.size`. This is the number of documents that are pulled back with every request ([Configuration | Elasticsearch for Apache Hadoop [8.0] | Elastic](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html#_advanced_settings)). You didn't mention which version you are using, but the default was 50 until 8.0. We changed it to 1000 because 50 is too low for a lot of use cases. Setting `es.scroll.size` higher will use more memory, but that's probably an acceptable tradeoff. If you are using an es-hadoop version earlier than 8.0 try setting it to 1000. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 22, 2022, 3:04pm UTC](https://discuss.elastic.co/t/reading-from-elasticsearch-index-using-spark-es-hadoop-connectors/297590/3 "2022-03-22T15:04:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.