# Spark, analogy between shard and partition is wrong
**URL:** https://discuss.elastic.co/t/spark-analogy-between-shard-and-partition-is-wrong/81847
**Category:** Elasticsearch
**Tags:** es-hadoop
**Created:** [April 10, 2017, 4:16pm UTC](https://discuss.elastic.co/t/spark-analogy-between-shard-and-partition-is-wrong/81847 "2017-04-10T16:16:47Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@ebuildy](https://discuss.elastic.co/u/ebuildy)
#### Post date: [April 10, 2017, 4:16pm UTC](https://discuss.elastic.co/t/spark-analogy-between-shard-and-partition-is-wrong/81847/1 "2017-04-10T16:16:47Z")
The plugin es4Hadoop make an analogy between ES shard and Spark partition, but it's not good.
Most of Spark users, advice to have "small" partitions, about 512M, for instance, Spark cannot cache in disk more than 2Gb (a know issue, not fix yet), whereas, elasticsearch advice to have big shard (10 Go), this is not compatible.
We should be able to control how many partitions when reading from elasticsearch. And by default, you should use the same value then HDFS use (64M/128M == 1 partition), I think it's a good idea to follow that.
(repartition can help, but it's very resources consuming and can give OOM).
---
### Author:  [@james.baiera](https://discuss.elastic.co/u/james.baiera)
#### Post date: [April 13, 2017, 4:32pm UTC](https://discuss.elastic.co/t/spark-analogy-between-shard-and-partition-is-wrong/81847/2 "2017-04-13T16:32:08Z")
@ebuildy in 5.0 we introduced the parallel reader functionality to the connector. When reading from Elasticsearch on clusters running 5.0 and up, the connector will subdivide the shards into slices based on the number of documents in them. This is configurable using the `es.input.max.docs.per.partition` (default 100,000). This should let you split up the larger ES partitions into smaller Spark friendly partitions.
---
### Author:  [@ebuildy](https://discuss.elastic.co/u/ebuildy)
#### Post date: [April 26, 2017, 7:10am UTC](https://discuss.elastic.co/t/spark-analogy-between-shard-and-partition-is-wrong/81847/3 "2017-04-26T07:10:41Z")
Ha fantastic!
(Cannot believe that Spark still has the max 2GB bug 😕 😕 )
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 24, 2017, 7:21am UTC](https://discuss.elastic.co/t/spark-analogy-between-shard-and-partition-is-wrong/81847/4 "2017-05-24T07:21:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.