# Elasticsearch Cluster - Difference is storage usage between nodes **URL:** \ **Category:** Elasticsearch\ **Created:** [December 8, 2020, 4:57am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-difference-is-storage-usage-between-nodes/257934 "2020-12-08T04:57:53Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [December 8, 2020, 4:57am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-difference-is-storage-usage-between-nodes/257934/1 "2020-12-08T04:57:53Z")
Hi Im having 3 node cluster with / partition has 32 GB. ES data folder is /var/lib/elasticsearch which is below / partition. Before I run load testing, I had 8 Million records and "used storage" details is below node 1 - 40% of / is used node 2 - 40% of / is used node 3 - 25% of / is used Then I ran load testing, after that it pumped 20 Million records taking total record count to 28 Million. After that storage node 1 - 71% of / is used node 2 - 72% of / is used node 3 - 28% of / is used As you see, node 1 & node 2 has jump in storage usage after load testing. But node-3 doesnt have much change in storage. I assumed all three nodes will have same number of records physically? I dont see any issue in data count when I run n Dev Tools. But storage used in not even in all 3 nodes Is it expected? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 8, 2020, 6:23am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-difference-is-storage-usage-between-nodes/257934/2 "2020-12-08T06:23:57Z")
What is the output of the [cat shards API](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/cat-shards.html)? Elasticsearch stores data in shards and this is the unit used for data distribution. If you have only one primary and one replica shard per index, that index data will only be stored on two nodes as the total number of shards is 2. If you have one index that is much larger than the others and it has only two shards seeing this kind of imbalance is not surprising. ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [December 8, 2020, 6:41am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-difference-is-storage-usage-between-nodes/257934/3 "2020-12-08T06:41:17Z")
> [@Christian\_Dahlqvist](#): > > one primary and one replica shard per index, that index data will only be stored on two nodes as the total number of shards is 2. If you have one index that is much larger than the others and it has only two shards seeing this kind of imbalance is not surprising. yeah I guess. Im creating index as below curl -s -X PUT $elasticsearchURL/$indexName?pretty -H 'Content-Type: application/json' -d '{"settings":{"index":{"number\_of\_shards" : 1,"number\_of\_replicas" : 1}}}' So will this have any impact in functionality?? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 8, 2020, 6:45am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-difference-is-storage-usage-between-nodes/257934/4 "2020-12-08T06:45:26Z")
If you want the cluster to be able to distribute data evenly you will likely need to set `number_of_shards` to `3`. ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [December 8, 2020, 6:46am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-difference-is-storage-usage-between-nodes/257934/5 "2020-12-08T06:46:36Z")
Ok. Will it have performance impact or data loss in any scenario? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 8, 2020, 6:50am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-difference-is-storage-usage-between-nodes/257934/6 "2020-12-08T06:50:58Z")
Given the size of your cluster I do not think that is likely. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 8, 2020, 7:07am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-difference-is-storage-usage-between-nodes/257934/7 "2020-12-08T07:07:03Z")
We recently added [docs that answer exactly this question](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#disk-based-shard-allocation) - see the section that starts > **TIP** : It is normal for the nodes in your cluster to be using very different amounts of disk space.... ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [December 8, 2020, 7:20am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-difference-is-storage-usage-between-nodes/257934/8 "2020-12-08T07:20:53Z")
Thank you for quick turnaround. Can I increase number of shards in existing cluster from 1 to 3? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 5, 2021, 7:21am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-difference-is-storage-usage-between-nodes/257934/9 "2021-01-05T07:21:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.