# Read from spark big index and memory usage **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [March 3, 2022, 7:43am UTC](https://discuss.elastic.co/t/read-from-spark-big-index-and-memory-usage/298707 "2022-03-03T07:43:37Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\ **Post date:** [March 3, 2022, 7:43am UTC](https://discuss.elastic.co/t/read-from-spark-big-index-and-memory-usage/298707/1 "2022-03-03T07:43:37Z")
We use Spark to read a full index of 80Go then write data as a parquet file. I see memory consumption is going up until 24G per Spark worker. Hence my question: **during the scroll search (1 hour), is the data stored in JVM heap on es4Hadoop side?** ---
**Author:** ![Keith\_Massey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keith_massey/32/83666_2.png) [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)\ **Post date:** [March 3, 2022, 2:43pm UTC](https://discuss.elastic.co/t/read-from-spark-big-index-and-memory-usage/298707/2 "2022-03-03T14:43:13Z")
We store one batch of documents (the number of documents specified in `es.scroll.size`) in heap memory in es-hadoop. That's typically nowhere near 24G. Without having seen your code, here are some thoughts: - What are you specifying for your executor memory size? Is it possible that the garbage collector is just waiting until you've used all the memory? - What is your batch size (`es.scroll.size`)? The default went from 50 to 1000 in 8.0, so if you just recently upgraded to 8.0 you might be seeing more memory use (and better performance) than before. If your average document size is unusually large you might want to reduce that. - Is it possible you're not letting go of memory somewhere? Can you reproduce the memory leak in a code snippet you can paste here? ---
**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\ **Post date:** [March 3, 2022, 8:15pm UTC](https://discuss.elastic.co/t/read-from-spark-big-index-and-memory-usage/298707/3 "2022-03-03T20:15:35Z")
Well my code is pretty straightforward, read ES then write parquet. Our Elasticsearch index is a 3 shards index of 80Go, with 70.000.000 docs. Spark run on YARN, 3 executors of 4 cores / 22Go memory, during 1 hour. I tune es4Hadoop: ```auto es.scroll.size=4000 es.input.max.docs.per.partition=5000000 es.scroll.keepalive=60m ``` So this gives 12 tasks, that runs all in // . We monitor this with the very nice elastic APM agent: ![Capture d’écran 2022-03-03 à 21.12.31](https://us1.discourse-cdn.com/elastic/original/3X/b/9/b913569142529f1b46c564d6f858120bd9648132.png) I was just wondering what is in the 20G heap and where go the ES data during the long search operation? So I am going to test with ```auto es.scroll.size=500 ``` ---
**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\ **Post date:** [March 4, 2022, 6:58am UTC](https://discuss.elastic.co/t/read-from-spark-big-index-and-memory-usage/298707/4 "2022-03-04T06:58:13Z")
Woa! I can see the heap is totally different with `scroll.size=500` : ![Capture d’écran 2022-03-04 à 07.55.44](https://us1.discourse-cdn.com/elastic/original/3X/2/b/2b982c758c267ac15fbeec4185e4f2fa77620e23.png) but it took 2 hours, so I have to figure out the best parameters to deal with - time duration - search // with `es.input.max.docs.per.partition` - heap with `scroll.size` Thanks you for the clear explanation! ---
**Author:** ![Keith\_Massey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keith_massey/32/83666_2.png) [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)\ **Post date:** [March 4, 2022, 3:15pm UTC](https://discuss.elastic.co/t/read-from-spark-big-index-and-memory-usage/298707/5 "2022-03-04T15:15:33Z")
I'm surprised it takes that long (although I don't know anything about your hardware or data, so maybe it's expected). Have you tried just reading all the data from Elasticsearch and then dropping it, to isolate if the slowness is reading from Elasticsearch or writing to parquet? My assumption is that the reading from Elasticsearch is the slow part, but I could be wrong. ---
**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\ **Post date:** [March 4, 2022, 3:37pm UTC](https://discuss.elastic.co/t/read-from-spark-big-index-and-memory-usage/298707/6 "2022-03-04T15:37:44Z")
I will on monday. Or maybe, is there any benchmark tools I could run to test reading ES from Spark? So I could compare results. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 1, 2022, 3:38pm UTC](https://discuss.elastic.co/t/read-from-spark-big-index-and-memory-usage/298707/7 "2022-04-01T15:38:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.