# ES/Spark RDD Partitioning Strangeness
**URL:** https://discuss.elastic.co/t/es-spark-rdd-partitioning-strangeness/238312
**Category:** Elasticsearch
**Tags:** es-hadoop
**Created:** [June 23, 2020, 5:12pm UTC](https://discuss.elastic.co/t/es-spark-rdd-partitioning-strangeness/238312 "2020-06-23T17:12:54Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@ikiril01](https://discuss.elastic.co/u/ikiril01)
#### Post date: [June 23, 2020, 5:12pm UTC](https://discuss.elastic.co/t/es-spark-rdd-partitioning-strangeness/238312/1 "2020-06-23T17:12:54Z")
We're using the es-hadoop connector to dump data from an ES index to spark (w/ pyspark, newAPIHadoopRDD). The issue we're seeing is that the RDD that is returned is skewed in terms of partitions - it always returns 9 partitions, with only 3 of them being populated.
Is there any way to have newAPIHadoopRDD return more evenly distributed partitions from the ES data? I realize that I can repartition them after the fact, but the initial set of partitions is causing some performance headaches.
---
### Author:  [@wangqinghuan](https://discuss.elastic.co/u/wangqinghuan)
#### Post date: [June 24, 2020, 2:34am UTC](https://discuss.elastic.co/t/es-spark-rdd-partitioning-strangeness/238312/2 "2020-06-24T02:34:03Z")
Welcome, Ivan.
You can control the number of documents per partition by set [es.input.max.docs.per.partition](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html) value.
---
### Author:  [@ikiril01](https://discuss.elastic.co/u/ikiril01)
#### Post date: [June 24, 2020, 1:11pm UTC](https://discuss.elastic.co/t/es-spark-rdd-partitioning-strangeness/238312/3 "2020-06-24T13:11:13Z")
Thanks Wang! That sounds like exactly what I need. I'll give it a shot.
---
### Author:  [@ikiril01](https://discuss.elastic.co/u/ikiril01)
#### Post date: [June 24, 2020, 4:33pm UTC](https://discuss.elastic.co/t/es-spark-rdd-partitioning-strangeness/238312/4 "2020-06-24T16:33:13Z")
I can confirm that **es.input.max.docs.per.partition** works well and solved my issue. Thanks again!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 22, 2020, 4:33pm UTC](https://discuss.elastic.co/t/es-spark-rdd-partitioning-strangeness/238312/5 "2020-07-22T16:33:25Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.