# Elasticsearch Shard distribution size differs enormously **URL:** \ **Category:** Elasticsearch\ **Created:** [August 9, 2021, 5:09pm UTC](https://discuss.elastic.co/t/elasticsearch-shard-distribution-size-differs-enormously/280845 "2021-08-09T17:09:37Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![sauasmast](https://avatars.discourse-cdn.com/v4/letter/s/67e7ee/32.png) [@sauasmast](https://discuss.elastic.co/u/sauasmast)\ **Post date:** [August 9, 2021, 5:09pm UTC](https://discuss.elastic.co/t/elasticsearch-shard-distribution-size-differs-enormously/280845/1 "2021-08-09T17:09:37Z")
I need to load 1.2 billion documents in the elasticsearch. As of today we have 6 nodes in the cluster. To equally distribute the shards among the 6 nodes I have mentioned the number of shards to be 42. I use spark and it takes me almost 3 days load the index. The shards distribution looks so off. ![Picture](https://us1.discourse-cdn.com/elastic/original/3X/3/d/3d0ca1dbb6ac419355c553fd9dbb8551278f0f57.png) The node6 only has two shards in it while node 2 has almost 10 shards. The size distribution is also not even. Some shards are 114.6gb while some are just 870mb within the same node. I have tried to figure out the solution too. I can include the `index.routing.allocation.total_shards_per_node: 7` while creating the index and make it evenly distribute. Will forcing the designated amount of shards in the node, crash the node if there is not enough resource available? I want to size the shards evenly. My index size is 900 gb apprx. I want each shards to be atleast 20 gb. Could I use the following setting while creating the index? `max_primary_shard_size: 25gb` Is setting up max shard size only possible through ilm policy and will I require roll over policy for that ? I am not too familiar with the ilm. Sorry if this does not make sense. The main reason I am trying to optimize the index is because I am getting timeout error on my application when I am querying the elastic search. I know I can increase my timeout time in my application and do some query optimization, but first I want to optimize my index and make my application as fast as possible. I load the index only one time and do not write any documents to it after onetime load. For additional data, which i load every 15 days, I create a different index and use an alias name on the both the indexes to query. Other than sharding if there is any suggestion to optimize my indexes I will really appreciate it. It takes me 3 days just to load the data so it is quite difficult to experiment. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 10, 2021, 3:40am UTC](https://discuss.elastic.co/t/elasticsearch-shard-distribution-size-differs-enormously/280845/2 "2021-08-10T03:40:00Z")
Welcome to our community! 😃 (I think you asked this on stackoverflow?) What does your indexing process look like? Are you indexing with a custom routing value? ---
**Author:** ![sauasmast](https://avatars.discourse-cdn.com/v4/letter/s/67e7ee/32.png) [@sauasmast](https://discuss.elastic.co/u/sauasmast)\ **Post date:** [August 10, 2021, 3:09pm UTC](https://discuss.elastic.co/t/elasticsearch-shard-distribution-size-differs-enormously/280845/3 "2021-08-10T15:09:44Z")
So sorry about that. I have followed up in the stack overflow. You can close this topic over here and we can interact in the stackoverflow. Link for the discussion on stackoverflow is as below: > ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 7, 2021, 3:10pm UTC](https://discuss.elastic.co/t/elasticsearch-shard-distribution-size-differs-enormously/280845/4 "2021-09-07T15:10:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.