# Issue with shard replicas being different in size **URL:** \ **Category:** Elasticsearch\ **Created:** [July 23, 2010, 9:32pm UTC](https://discuss.elastic.co/t/issue-with-shard-replicas-being-different-in-size/3144 "2010-07-23T21:32:55Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Grant\_Rodgers](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grant_rodgers/32/3204_2.png) [@Grant\_Rodgers](https://discuss.elastic.co/u/Grant_Rodgers)\ **Post date:** [July 23, 2010, 9:32pm UTC](https://discuss.elastic.co/t/issue-with-shard-replicas-being-different-in-size/3144/1 "2010-07-23T21:32:55Z")
Yesterday I did a full reindex of an index with ~8 million documents. I had 3 nodes on EC2 large instances. There were 8 clients indexing documents at the rate of ~1100 documents per minute per client. Today I did some match\_all count queries to verify the index size, and I notice that every other result is different: sometimes 5m, sometimes 8m. I look at the index status and I see that two shards have very differently sized replicas: The primaries have \<120k documents, and the replicas have \>1.3m documents. I tried stopping nodes to see if I could get the correct replicas to become the primaries, but it seems they may be stored in the gateway as the much smaller size rather than the correct size. It appears the behavior is that if a replica goes down, it is restored from the primary, but if a primary goes down, it is restored from the gateway. Here is the output from the index status command. Notice that shard 4 has a big difference between the size of the replica and the primary. Shard 3 originally had the same issue, but the correct replica was lost when the node with that replica was accidentally restarted (oops!) { "ok" : true, "\_shards" : { "total" : 12, "successful" : 12, "failed" : 0 }, "indices" : { "stats" : { "aliases" : [], "settings" : { "index.number\_of\_shards" : "6", "index.number\_of\_replicas" : "1" }, "store\_size" : "20.3g", "store\_size\_in\_bytes" : 21814342869, "estimated\_flushable\_memory\_size" : "96.5m", "estimated\_flushable\_memory\_size\_in\_bytes" : 101234949, "translog\_operations" : 102062, "docs" : { "num\_docs" : 5738311, "max\_doc" : 5744803, "deleted\_docs" : 6492 }, "shards" : { "0" : [ { "routing" : { "state" : "STARTED", "primary" : false, "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", "relocating\_node" : null, "shard" : 0, "index" : "stats" }, "state" : "STARTED", "store\_size" : "2.6g", "store\_size\_in\_bytes" : 2842186930, "estimated\_flushable\_memory\_size" : "298.8k", "estimated\_flushable\_memory\_size\_in\_bytes" : 306021, "translog\_id" : 0, "translog\_operations" : 306, "docs" : { "num\_docs" : 1352920, "max\_doc" : 1352920, "deleted\_docs" : 0 } }, { "routing" : { "state" : "RELOCATING", "primary" : true, "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", "relocating\_node" : "411f522e- bfbc-4e0b-8a70-6451d1bef1da", "shard" : 0, "index" : "stats" }, "state" : "STARTED", "store\_size" : "2.6g", "store\_size\_in\_bytes" : 2842196644, "estimated\_flushable\_memory\_size" : "298.8k", "estimated\_flushable\_memory\_size\_in\_bytes" : 306021, "translog\_id" : 1, "translog\_operations" : 306, "docs" : { "num\_docs" : 1352920, "max\_doc" : 1352920, "deleted\_docs" : 0 } } ], "1" : [ { "routing" : { "state" : "STARTED", "primary" : false, "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", "relocating\_node" : null, "shard" : 1, "index" : "stats" }, "state" : "STARTED", "store\_size" : "1.9g", "store\_size\_in\_bytes" : 2109353140, "estimated\_flushable\_memory\_size" : "304.8k", "estimated\_flushable\_memory\_size\_in\_bytes" : 312123, "translog\_id" : 0, "translog\_operations" : 323, "docs" : { "num\_docs" : 1413888, "max\_doc" : 1418829, "deleted\_docs" : 4941 } }, { "routing" : { "state" : "RELOCATING", "primary" : true, "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", "relocating\_node" : "411f522e- bfbc-4e0b-8a70-6451d1bef1da", "shard" : 1, "index" : "stats" }, "state" : "STARTED", "store\_size" : "1.9g", "store\_size\_in\_bytes" : 2109369148, "estimated\_flushable\_memory\_size" : "304.8k", "estimated\_flushable\_memory\_size\_in\_bytes" : 312123, "translog\_id" : 1, "translog\_operations" : 323, "docs" : { "num\_docs" : 1413888, "max\_doc" : 1418829, "deleted\_docs" : 4941 } } ], "2" : [ { "routing" : { "state" : "RELOCATING", "primary" : false, "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", "relocating\_node" : "411f522e- bfbc-4e0b-8a70-6451d1bef1da", "shard" : 2, "index" : "stats" }, "state" : "STARTED", "store\_size" : "1.9g", "store\_size\_in\_bytes" : 2050764870, "estimated\_flushable\_memory\_size" : "877.4k", "estimated\_flushable\_memory\_size\_in\_bytes" : 898541, "translog\_id" : 0, "translog\_operations" : 897, "docs" : { "num\_docs" : 1412645, "max\_doc" : 1413300, "deleted\_docs" : 655 } }, { "routing" : { "state" : "STARTED", "primary" : true, "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", "relocating\_node" : null, "shard" : 2, "index" : "stats" }, "state" : "STARTED", "store\_size" : "1.9g", "store\_size\_in\_bytes" : 2050751585, "estimated\_flushable\_memory\_size" : "876.7k", "estimated\_flushable\_memory\_size\_in\_bytes" : 897826, "translog\_id" : 0, "translog\_operations" : 896, "docs" : { "num\_docs" : 1412644, "max\_doc" : 1413299, "deleted\_docs" : 655 } } ], "3" : [ { "routing" : { "state" : "RELOCATING", "primary" : true, "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", "relocating\_node" : "411f522e- bfbc-4e0b-8a70-6451d1bef1da", "shard" : 3, "index" : "stats" }, "state" : "STARTED", "store\_size" : "195m", "store\_size\_in\_bytes" : 204574035, "estimated\_flushable\_memory\_size" : "3.7m", "estimated\_flushable\_memory\_size\_in\_bytes" : 3949835, "translog\_id" : 0, "translog\_operations" : 3842, "docs" : { "num\_docs" : 103921, "max\_doc" : 103921, "deleted\_docs" : 0 } }, { "routing" : { "state" : "STARTED", "primary" : false, "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", "relocating\_node" : null, "shard" : 3, "index" : "stats" }, "state" : "STARTED", "store\_size" : "195m", "store\_size\_in\_bytes" : 204572396, "estimated\_flushable\_memory\_size" : "3.7m", "estimated\_flushable\_memory\_size\_in\_bytes" : 3949835, "translog\_id" : 0, "translog\_operations" : 3842, "docs" : { "num\_docs" : 103921, "max\_doc" : 103921, "deleted\_docs" : 0 } } ], "4" : [ { "routing" : { "state" : "STARTED", "primary" : false, "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", "relocating\_node" : null, "shard" : 4, "index" : "stats" }, "state" : "STARTED", "store\_size" : "1.9g", "store\_size\_in\_bytes" : 2106604638, "estimated\_flushable\_memory\_size" : "40.9m", "estimated\_flushable\_memory\_size\_in\_bytes" : 42929689, "translog\_id" : 262, "translog\_operations" : 43410, "docs" : { "num\_docs" : 1415174, "max\_doc" : 1419993, "deleted\_docs" : 4819 } }, { "routing" : { "state" : "STARTED", "primary" : true, "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", "relocating\_node" : null, "shard" : 4, "index" : "stats" }, "state" : "STARTED", "store\_size" : "104.9m", "store\_size\_in\_bytes" : 110022541, "estimated\_flushable\_memory\_size" : "3.2m", "estimated\_flushable\_memory\_size\_in\_bytes" : 3392079, "translog\_id" : 0, "translog\_operations" : 3311, "docs" : { "num\_docs" : 38319, "max\_doc" : 38319, "deleted\_docs" : 0 } } ], "5" : [ { "routing" : { "state" : "STARTED", "primary" : true, "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", "relocating\_node" : null, "shard" : 5, "index" : "stats" }, "state" : "STARTED", "store\_size" : "2.8g", "store\_size\_in\_bytes" : 3074996380, "estimated\_flushable\_memory\_size" : "41.9m", "estimated\_flushable\_memory\_size\_in\_bytes" : 43980856, "translog\_id" : 276, "translog\_operations" : 44606, "docs" : { "num\_docs" : 1416619, "max\_doc" : 1417515, "deleted\_docs" : 896 } }, { "routing" : { "state" : "STARTED", "primary" : false, "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", "relocating\_node" : null, "shard" : 5, "index" : "stats" }, "state" : "STARTED", "store\_size" : "1.9g", "store\_size\_in\_bytes" : 2108950562, "estimated\_flushable\_memory\_size" : "0", "estimated\_flushable\_memory\_size\_in\_bytes" : 0, "translog\_id" : 1, "translog\_operations" : 0, "docs" : { "num\_docs" : 1416619, "max\_doc" : 1417515, "deleted\_docs" : 896 } } ] } } } ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 24, 2010, 9:56am UTC](https://discuss.elastic.co/t/issue-with-shard-replicas-being-different-in-size/3144/2 "2010-07-24T09:56:48Z")
Hi, ``` The problem of the primary, when shutdown, it gets allocated and ``` restored from the gateway (instead of having one of the replicas become the primary) is fixed in 0.9. The difference between the two is strange (replica and primary). I have fixed in 0.9 some cases where this might happen during relocation of shards, did that happen to you? Also, are you using the s3 gateway? -shay.banon On Sat, Jul 24, 2010 at 12:32 AM, Grant Rodgers [grantr@gmail.com](mailto:grantr@gmail.com) wrote: > Yesterday I did a full reindex of an index with ~8 million documents. > I had 3 nodes on EC2 large instances. There were 8 clients indexing > documents at the rate of ~1100 documents per minute per client. > > Today I did some match\_all count queries to verify the index size, and > I notice that every other result is different: sometimes 5m, sometimes > 8m. I look at the index status and I see that two shards have very > differently sized replicas: The primaries have \<120k documents, and > the replicas have \>1.3m documents. > > I tried stopping nodes to see if I could get the correct replicas to > become the primaries, but it seems they may be stored in the gateway > as the much smaller size rather than the correct size. It appears the > behavior is that if a replica goes down, it is restored from the > primary, but if a primary goes down, it is restored from the gateway. > > Here is the output from the index status command. Notice that shard 4 > has a big difference between the size of the replica and the primary. > Shard 3 originally had the same issue, but the correct replica was > lost when the node with that replica was accidentally restarted > (oops!) > > { > "ok" : true, > "\_shards" : { > "total" : 12, > "successful" : 12, > "failed" : 0 > }, > "indices" : { > "stats" : { > "aliases" : , > "settings" : { > "index.number\_of\_shards" : "6", > "index.number\_of\_replicas" : "1" > }, > "store\_size" : "20.3g", > "store\_size\_in\_bytes" : 21814342869, > "estimated\_flushable\_memory\_size" : "96.5m", > "estimated\_flushable\_memory\_size\_in\_bytes" : 101234949, > "translog\_operations" : 102062, > "docs" : { > "num\_docs" : 5738311, > "max\_doc" : 5744803, > "deleted\_docs" : 6492 > }, > "shards" : { > "0" : [ { > "routing" : { > "state" : "STARTED", > "primary" : false, > "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", > "relocating\_node" : null, > "shard" : 0, > "index" : "stats" > }, > "state" : "STARTED", > "store\_size" : "2.6g", > "store\_size\_in\_bytes" : 2842186930, > "estimated\_flushable\_memory\_size" : "298.8k", > "estimated\_flushable\_memory\_size\_in\_bytes" : 306021, > "translog\_id" : 0, > "translog\_operations" : 306, > "docs" : { > "num\_docs" : 1352920, > "max\_doc" : 1352920, > "deleted\_docs" : 0 > } > }, { > "routing" : { > "state" : "RELOCATING", > "primary" : true, > "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", > "relocating\_node" : "411f522e- > bfbc-4e0b-8a70-6451d1bef1da", > "shard" : 0, > "index" : "stats" > }, > "state" : "STARTED", > "store\_size" : "2.6g", > "store\_size\_in\_bytes" : 2842196644, > "estimated\_flushable\_memory\_size" : "298.8k", > "estimated\_flushable\_memory\_size\_in\_bytes" : 306021, > "translog\_id" : 1, > "translog\_operations" : 306, > "docs" : { > "num\_docs" : 1352920, > "max\_doc" : 1352920, > "deleted\_docs" : 0 > } > } ], > "1" : [ { > "routing" : { > "state" : "STARTED", > "primary" : false, > "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", > "relocating\_node" : null, > "shard" : 1, > "index" : "stats" > }, > "state" : "STARTED", > "store\_size" : "1.9g", > "store\_size\_in\_bytes" : 2109353140, > "estimated\_flushable\_memory\_size" : "304.8k", > "estimated\_flushable\_memory\_size\_in\_bytes" : 312123, > "translog\_id" : 0, > "translog\_operations" : 323, > "docs" : { > "num\_docs" : 1413888, > "max\_doc" : 1418829, > "deleted\_docs" : 4941 > } > }, { > "routing" : { > "state" : "RELOCATING", > "primary" : true, > "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", > "relocating\_node" : "411f522e- > bfbc-4e0b-8a70-6451d1bef1da", > "shard" : 1, > "index" : "stats" > }, > "state" : "STARTED", > "store\_size" : "1.9g", > "store\_size\_in\_bytes" : 2109369148, > "estimated\_flushable\_memory\_size" : "304.8k", > "estimated\_flushable\_memory\_size\_in\_bytes" : 312123, > "translog\_id" : 1, > "translog\_operations" : 323, > "docs" : { > "num\_docs" : 1413888, > "max\_doc" : 1418829, > "deleted\_docs" : 4941 > } > } ], > "2" : [ { > "routing" : { > "state" : "RELOCATING", > "primary" : false, > "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", > "relocating\_node" : "411f522e- > bfbc-4e0b-8a70-6451d1bef1da", > "shard" : 2, > "index" : "stats" > }, > "state" : "STARTED", > "store\_size" : "1.9g", > "store\_size\_in\_bytes" : 2050764870, > "estimated\_flushable\_memory\_size" : "877.4k", > "estimated\_flushable\_memory\_size\_in\_bytes" : 898541, > "translog\_id" : 0, > "translog\_operations" : 897, > "docs" : { > "num\_docs" : 1412645, > "max\_doc" : 1413300, > "deleted\_docs" : 655 > } > }, { > "routing" : { > "state" : "STARTED", > "primary" : true, > "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", > "relocating\_node" : null, > "shard" : 2, > "index" : "stats" > }, > "state" : "STARTED", > "store\_size" : "1.9g", > "store\_size\_in\_bytes" : 2050751585, > "estimated\_flushable\_memory\_size" : "876.7k", > "estimated\_flushable\_memory\_size\_in\_bytes" : 897826, > "translog\_id" : 0, > "translog\_operations" : 896, > "docs" : { > "num\_docs" : 1412644, > "max\_doc" : 1413299, > "deleted\_docs" : 655 > } > } ], > "3" : [ { > "routing" : { > "state" : "RELOCATING", > "primary" : true, > "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", > "relocating\_node" : "411f522e- > bfbc-4e0b-8a70-6451d1bef1da", > "shard" : 3, > "index" : "stats" > }, > "state" : "STARTED", > "store\_size" : "195m", > "store\_size\_in\_bytes" : 204574035, > "estimated\_flushable\_memory\_size" : "3.7m", > "estimated\_flushable\_memory\_size\_in\_bytes" : 3949835, > "translog\_id" : 0, > "translog\_operations" : 3842, > "docs" : { > "num\_docs" : 103921, > "max\_doc" : 103921, > "deleted\_docs" : 0 > } > }, { > "routing" : { > "state" : "STARTED", > "primary" : false, > "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", > "relocating\_node" : null, > "shard" : 3, > "index" : "stats" > }, > "state" : "STARTED", > "store\_size" : "195m", > "store\_size\_in\_bytes" : 204572396, > "estimated\_flushable\_memory\_size" : "3.7m", > "estimated\_flushable\_memory\_size\_in\_bytes" : 3949835, > "translog\_id" : 0, > "translog\_operations" : 3842, > "docs" : { > "num\_docs" : 103921, > "max\_doc" : 103921, > "deleted\_docs" : 0 > } > } ], > "4" : [ { > "routing" : { > "state" : "STARTED", > "primary" : false, > "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", > "relocating\_node" : null, > "shard" : 4, > "index" : "stats" > }, > "state" : "STARTED", > "store\_size" : "1.9g", > "store\_size\_in\_bytes" : 2106604638, > "estimated\_flushable\_memory\_size" : "40.9m", > "estimated\_flushable\_memory\_size\_in\_bytes" : 42929689, > "translog\_id" : 262, > "translog\_operations" : 43410, > "docs" : { > "num\_docs" : 1415174, > "max\_doc" : 1419993, > "deleted\_docs" : 4819 > } > }, { > "routing" : { > "state" : "STARTED", > "primary" : true, > "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", > "relocating\_node" : null, > "shard" : 4, > "index" : "stats" > }, > "state" : "STARTED", > "store\_size" : "104.9m", > "store\_size\_in\_bytes" : 110022541, > "estimated\_flushable\_memory\_size" : "3.2m", > "estimated\_flushable\_memory\_size\_in\_bytes" : 3392079, > "translog\_id" : 0, > "translog\_operations" : 3311, > "docs" : { > "num\_docs" : 38319, > "max\_doc" : 38319, > "deleted\_docs" : 0 > } > } ], > "5" : [ { > "routing" : { > "state" : "STARTED", > "primary" : true, > "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", > "relocating\_node" : null, > "shard" : 5, > "index" : "stats" > }, > "state" : "STARTED", > "store\_size" : "2.8g", > "store\_size\_in\_bytes" : 3074996380, > "estimated\_flushable\_memory\_size" : "41.9m", > "estimated\_flushable\_memory\_size\_in\_bytes" : 43980856, > "translog\_id" : 276, > "translog\_operations" : 44606, > "docs" : { > "num\_docs" : 1416619, > "max\_doc" : 1417515, > "deleted\_docs" : 896 > } > }, { > "routing" : { > "state" : "STARTED", > "primary" : false, > "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", > "relocating\_node" : null, > "shard" : 5, > "index" : "stats" > }, > "state" : "STARTED", > "store\_size" : "1.9g", > "store\_size\_in\_bytes" : 2108950562, > "estimated\_flushable\_memory\_size" : "0", > "estimated\_flushable\_memory\_size\_in\_bytes" : 0, > "translog\_id" : 1, > "translog\_operations" : 0, > "docs" : { > "num\_docs" : 1416619, > "max\_doc" : 1417515, > "deleted\_docs" : 896 > } > } ] > } > } > } ---
**Author:** ![Grant\_Rodgers](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grant_rodgers/32/3204_2.png) [@Grant\_Rodgers](https://discuss.elastic.co/u/Grant_Rodgers)\ **Post date:** [July 24, 2010, 7:32pm UTC](https://discuss.elastic.co/t/issue-with-shard-replicas-being-different-in-size/3144/3 "2010-07-24T19:32:31Z")
Yes, actually, after the index we also replaced two of the nodes. The procedure we used was to stop one node, wait for the shards to stabilize on the other two, then start the new one, and wait for the shards to stabilize again. We did this twice, so there was a lot of relocating going on. Unfortunately I didn't test the numbers before the node swaps, but we are doing a full reindex this weekend so we will see if the issue comes up again without them. We are using 0.8.0 and the s3 gateway. We're looking forward to 0.9 quite keenly. The improved shard management mentioned here, plus work directory reuse, new cloud gateway, and new facets are all things we can use! Any idea of a timetable for the release? Thanks, Grant On Jul 24, 2:56 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Hi, > > ``` > The problem of the primary, when shutdown, it gets allocated and > > ``` > > restored from the gateway (instead of having one of the replicas become the > primary) is fixed in 0.9. The difference between the two is strange (replica > and primary). I have fixed in 0.9 some cases where this might happen during > relocation of shards, did that happen to you? Also, are you using the s3 > gateway? > > -shay.banon > > On Sat, Jul 24, 2010 at 12:32 AM, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com) wrote: > > > Yesterday I did a full reindex of an index with ~8 million documents. > > I had 3 nodes on EC2 large instances. There were 8 clients indexing > > documents at the rate of ~1100 documents per minute per client. > > > Today I did some match\_all count queries to verify the index size, and > > I notice that every other result is different: sometimes 5m, sometimes > > 8m. I look at the index status and I see that two shards have very > > differently sized replicas: The primaries have \<120k documents, and > > the replicas have \>1.3m documents. > > > I tried stopping nodes to see if I could get the correct replicas to > > become the primaries, but it seems they may be stored in the gateway > > as the much smaller size rather than the correct size. It appears the > > behavior is that if a replica goes down, it is restored from the > > primary, but if a primary goes down, it is restored from the gateway. > > > Here is the output from the index status command. Notice that shard 4 > > has a big difference between the size of the replica and the primary. > > Shard 3 originally had the same issue, but the correct replica was > > lost when the node with that replica was accidentally restarted > > (oops!) > > > { > > "ok" : true, > > "\_shards" : { > > "total" : 12, > > "successful" : 12, > > "failed" : 0 > > }, > > "indices" : { > > "stats" : { > > "aliases" : , > > "settings" : { > > "index.number\_of\_shards" : "6", > > "index.number\_of\_replicas" : "1" > > }, > > "store\_size" : "20.3g", > > "store\_size\_in\_bytes" : 21814342869, > > "estimated\_flushable\_memory\_size" : "96.5m", > > "estimated\_flushable\_memory\_size\_in\_bytes" : 101234949, > > "translog\_operations" : 102062, > > "docs" : { > > "num\_docs" : 5738311, > > "max\_doc" : 5744803, > > "deleted\_docs" : 6492 > > }, > > "shards" : { > > "0" : [ { > > "routing" : { > > "state" : "STARTED", > > "primary" : false, > > "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", > > "relocating\_node" : null, > > "shard" : 0, > > "index" : "stats" > > }, > > "state" : "STARTED", > > "store\_size" : "2.6g", > > "store\_size\_in\_bytes" : 2842186930, > > "estimated\_flushable\_memory\_size" : "298.8k", > > "estimated\_flushable\_memory\_size\_in\_bytes" : 306021, > > "translog\_id" : 0, > > "translog\_operations" : 306, > > "docs" : { > > "num\_docs" : 1352920, > > "max\_doc" : 1352920, > > "deleted\_docs" : 0 > > } > > }, { > > "routing" : { > > "state" : "RELOCATING", > > "primary" : true, > > "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", > > "relocating\_node" : "411f522e- > > bfbc-4e0b-8a70-6451d1bef1da", > > "shard" : 0, > > "index" : "stats" > > }, > > "state" : "STARTED", > > "store\_size" : "2.6g", > > "store\_size\_in\_bytes" : 2842196644, > > "estimated\_flushable\_memory\_size" : "298.8k", > > "estimated\_flushable\_memory\_size\_in\_bytes" : 306021, > > "translog\_id" : 1, > > "translog\_operations" : 306, > > "docs" : { > > "num\_docs" : 1352920, > > "max\_doc" : 1352920, > > "deleted\_docs" : 0 > > } > > } ], > > "1" : [ { > > "routing" : { > > "state" : "STARTED", > > "primary" : false, > > "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", > > "relocating\_node" : null, > > "shard" : 1, > > "index" : "stats" > > }, > > "state" : "STARTED", > > "store\_size" : "1.9g", > > "store\_size\_in\_bytes" : 2109353140, > > "estimated\_flushable\_memory\_size" : "304.8k", > > "estimated\_flushable\_memory\_size\_in\_bytes" : 312123, > > "translog\_id" : 0, > > "translog\_operations" : 323, > > "docs" : { > > "num\_docs" : 1413888, > > "max\_doc" : 1418829, > > "deleted\_docs" : 4941 > > } > > }, { > > "routing" : { > > "state" : "RELOCATING", > > "primary" : true, > > "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", > > "relocating\_node" : "411f522e- > > bfbc-4e0b-8a70-6451d1bef1da", > > "shard" : 1, > > "index" : "stats" > > }, > > "state" : "STARTED", > > "store\_size" : "1.9g", > > "store\_size\_in\_bytes" : 2109369148, > > "estimated\_flushable\_memory\_size" : "304.8k", > > "estimated\_flushable\_memory\_size\_in\_bytes" : 312123, > > "translog\_id" : 1, > > "translog\_operations" : 323, > > "docs" : { > > "num\_docs" : 1413888, > > "max\_doc" : 1418829, > > "deleted\_docs" : 4941 > > } > > } ], > > "2" : [ { > > "routing" : { > > "state" : "RELOCATING", > > "primary" : false, > > "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", > > "relocating\_node" : "411f522e- > > bfbc-4e0b-8a70-6451d1bef1da", > > "shard" : 2, > > "index" : "stats" > > }, > > "state" : "STARTED", > > "store\_size" : "1.9g", > > "store\_size\_in\_bytes" : 2050764870, > > "estimated\_flushable\_memory\_size" : "877.4k", > > "estimated\_flushable\_memory\_size\_in\_bytes" : 898541, > > "translog\_id" : 0, > > "translog\_operations" : 897, > > "docs" : { > > "num\_docs" : 1412645, > > "max\_doc" : 1413300, > > "deleted\_docs" : 655 > > } > > }, { > > "routing" : { > > "state" : "STARTED", > > "primary" : true, > > "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", > > "relocating\_node" : null, > > "shard" : 2, > > "index" : "stats" > > }, > > "state" : "STARTED", > > "store\_size" : "1.9g", > > "store\_size\_in\_bytes" : 2050751585, > > "estimated\_flushable\_memory\_size" : "876.7k", > > "estimated\_flushable\_memory\_size\_in\_bytes" : 897826, > > "translog\_id" : 0, > > "translog\_operations" : 896, > > "docs" : { > > "num\_docs" : 1412644, > > "max\_doc" : 1413299, > > "deleted\_docs" : 655 > > } > > } ], > > "3" : [ { > > "routing" : { > > "state" : "RELOCATING", > > "primary" : true, > > "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", > > "relocating\_node" : "411f522e- > > bfbc-4e0b-8a70-6451d1bef1da", > > "shard" : 3, > > "index" : "stats" > > }, > > "state" : "STARTED", > > "store\_size" : "195m", > > "store\_size\_in\_bytes" : 204574035, > > "estimated\_flushable\_memory\_size" : "3.7m", > > "estimated\_flushable\_memory\_size\_in\_bytes" : 3949835, > > "translog\_id" : 0, > > "translog\_operations" : 3842, > > "docs" : { > > "num\_docs" : 103921, > > "max\_doc" : 103921, > > "deleted\_docs" : 0 > > } > > }, { > > "routing" : { > > "state" : "STARTED", > > "primary" : false, > > "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", > > "relocating\_node" : null, > > "shard" : 3, > > "index" : "stats" > > }, > > "state" : "STARTED", > > "store\_size" : "195m", > > "store\_size\_in\_bytes" : 204572396, > > "estimated\_flushable\_memory\_size" : "3.7m", > > "estimated\_flushable\_memory\_size\_in\_bytes" : 3949835, > > "translog\_id" : 0, > > "translog\_operations" : 3842, > > "docs" : { > > "num\_docs" : 103921, > > "max\_doc" : 103921, > > "deleted\_docs" : 0 > > } > > } ], > > "4" : [ { > > "routing" : { > > "state" : "STARTED", > > "primary" : false, > > "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", > > "relocating\_node" : null, > > "shard" : 4, > > "index" : "stats" > > }, > > "state" : "STARTED", > > "store\_size" : "1.9g", > > "store\_size\_in\_bytes" : 2106604638, > > "estimated\_flushable\_memory\_size" : "40.9m", > > "estimated\_flushable\_memory\_size\_in\_bytes" : 42929689, > > "translog\_id" : 262, > > "translog\_operations" : 43410, > > "docs" : { > > "num\_docs" : 1415174, > > "max\_doc" : 1419993, > > "deleted\_docs" : 4819 > > } > > }, { > > "routing" : { > > "state" : "STARTED", > > "primary" : true, > > "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", > > "relocating\_node" : null, > > "shard" : 4, > > "index" : "stats" > > }, > > "state" : "STARTED", > > "store\_size" : "104.9m", > > "store\_size\_in\_bytes" : 110022541, > > "estimated\_flushable\_memory\_size" : "3.2m", > > "estimated\_flushable\_memory\_size\_in\_bytes" : 3392079, > > "translog\_id" : 0, > > "translog\_operations" : 3311, > > "docs" : { > > "num\_docs" : 38319, > > "max\_doc" : 38319, > > "deleted\_docs" : 0 > > } > > } ], > > "5" : [ { > > "routing" : { > > "state" : "STARTED", > > "primary" : true, > > "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", > > "relocating\_node" : null, > > "shard" : 5, > > "index" : "stats" > > ... > > read more » ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 24, 2010, 7:47pm UTC](https://discuss.elastic.co/t/issue-with-shard-replicas-being-different-in-size/3144/4 "2010-07-24T19:47:28Z")
Hi, Can you wait with the reindexing? Almost done with the new AWS specific cloud plugin, once its done, will ping this mailing list with docs on how to use it. For now, decided to go with the formal AWS SDK to simplify things, though in the future might implement a proper non blocking myself. -shay.banon On Sat, Jul 24, 2010 at 10:32 PM, Grant Rodgers [grantr@gmail.com](mailto:grantr@gmail.com) wrote: > Yes, actually, after the index we also replaced two of the nodes. The > procedure we used was to stop one node, wait for the shards to > stabilize on the other two, then start the new one, and wait for the > shards to stabilize again. We did this twice, so there was a lot of > relocating going on. Unfortunately I didn't test the numbers before > the node swaps, but we are doing a full reindex this weekend so we > will see if the issue comes up again without them. We are using 0.8.0 > and the s3 gateway. > > We're looking forward to 0.9 quite keenly. The improved shard > management mentioned here, plus work directory reuse, new cloud > gateway, and new facets are all things we can use! Any idea of a > timetable for the release? > > Thanks, > Grant > > On Jul 24, 2:56 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Hi, > > > > ``` > > The problem of the primary, when shutdown, it gets allocated and > > > > ``` > > > > restored from the gateway (instead of having one of the replicas become > > the > > primary) is fixed in 0.9. The difference between the two is strange > > (replica > > and primary). I have fixed in 0.9 some cases where this might happen > > during > > relocation of shards, did that happen to you? Also, are you using the s3 > > gateway? > > > > -shay.banon > > > > On Sat, Jul 24, 2010 at 12:32 AM, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com) > > wrote: > > > > > Yesterday I did a full reindex of an index with ~8 million documents. > > > I had 3 nodes on EC2 large instances. There were 8 clients indexing > > > documents at the rate of ~1100 documents per minute per client. > > > > > Today I did some match\_all count queries to verify the index size, and > > > I notice that every other result is different: sometimes 5m, sometimes > > > 8m. I look at the index status and I see that two shards have very > > > differently sized replicas: The primaries have \<120k documents, and > > > the replicas have \>1.3m documents. > > > > > I tried stopping nodes to see if I could get the correct replicas to > > > become the primaries, but it seems they may be stored in the gateway > > > as the much smaller size rather than the correct size. It appears the > > > behavior is that if a replica goes down, it is restored from the > > > primary, but if a primary goes down, it is restored from the gateway. > > > > > Here is the output from the index status command. Notice that shard 4 > > > has a big difference between the size of the replica and the primary. > > > Shard 3 originally had the same issue, but the correct replica was > > > lost when the node with that replica was accidentally restarted > > > (oops!) > > > > > { > > > "ok" : true, > > > "\_shards" : { > > > "total" : 12, > > > "successful" : 12, > > > "failed" : 0 > > > }, > > > "indices" : { > > > "stats" : { > > > "aliases" : , > > > "settings" : { > > > "index.number\_of\_shards" : "6", > > > "index.number\_of\_replicas" : "1" > > > }, > > > "store\_size" : "20.3g", > > > "store\_size\_in\_bytes" : 21814342869, > > > "estimated\_flushable\_memory\_size" : "96.5m", > > > "estimated\_flushable\_memory\_size\_in\_bytes" : 101234949, > > > "translog\_operations" : 102062, > > > "docs" : { > > > "num\_docs" : 5738311, > > > "max\_doc" : 5744803, > > > "deleted\_docs" : 6492 > > > }, > > > "shards" : { > > > "0" : [ { > > > "routing" : { > > > "state" : "STARTED", > > > "primary" : false, > > > "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", > > > "relocating\_node" : null, > > > "shard" : 0, > > > "index" : "stats" > > > }, > > > "state" : "STARTED", > > > "store\_size" : "2.6g", > > > "store\_size\_in\_bytes" : 2842186930, > > > "estimated\_flushable\_memory\_size" : "298.8k", > > > "estimated\_flushable\_memory\_size\_in\_bytes" : 306021, > > > "translog\_id" : 0, > > > "translog\_operations" : 306, > > > "docs" : { > > > "num\_docs" : 1352920, > > > "max\_doc" : 1352920, > > > "deleted\_docs" : 0 > > > } > > > }, { > > > "routing" : { > > > "state" : "RELOCATING", > > > "primary" : true, > > > "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", > > > "relocating\_node" : "411f522e- > > > bfbc-4e0b-8a70-6451d1bef1da", > > > "shard" : 0, > > > "index" : "stats" > > > }, > > > "state" : "STARTED", > > > "store\_size" : "2.6g", > > > "store\_size\_in\_bytes" : 2842196644, > > > "estimated\_flushable\_memory\_size" : "298.8k", > > > "estimated\_flushable\_memory\_size\_in\_bytes" : 306021, > > > "translog\_id" : 1, > > > "translog\_operations" : 306, > > > "docs" : { > > > "num\_docs" : 1352920, > > > "max\_doc" : 1352920, > > > "deleted\_docs" : 0 > > > } > > > } ], > > > "1" : [ { > > > "routing" : { > > > "state" : "STARTED", > > > "primary" : false, > > > "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", > > > "relocating\_node" : null, > > > "shard" : 1, > > > "index" : "stats" > > > }, > > > "state" : "STARTED", > > > "store\_size" : "1.9g", > > > "store\_size\_in\_bytes" : 2109353140, > > > "estimated\_flushable\_memory\_size" : "304.8k", > > > "estimated\_flushable\_memory\_size\_in\_bytes" : 312123, > > > "translog\_id" : 0, > > > "translog\_operations" : 323, > > > "docs" : { > > > "num\_docs" : 1413888, > > > "max\_doc" : 1418829, > > > "deleted\_docs" : 4941 > > > } > > > }, { > > > "routing" : { > > > "state" : "RELOCATING", > > > "primary" : true, > > > "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", > > > "relocating\_node" : "411f522e- > > > bfbc-4e0b-8a70-6451d1bef1da", > > > "shard" : 1, > > > "index" : "stats" > > > }, > > > "state" : "STARTED", > > > "store\_size" : "1.9g", > > > "store\_size\_in\_bytes" : 2109369148, > > > "estimated\_flushable\_memory\_size" : "304.8k", > > > "estimated\_flushable\_memory\_size\_in\_bytes" : 312123, > > > "translog\_id" : 1, > > > "translog\_operations" : 323, > > > "docs" : { > > > "num\_docs" : 1413888, > > > "max\_doc" : 1418829, > > > "deleted\_docs" : 4941 > > > } > > > } ], > > > "2" : [ { > > > "routing" : { > > > "state" : "RELOCATING", > > > "primary" : false, > > > "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", > > > "relocating\_node" : "411f522e- > > > bfbc-4e0b-8a70-6451d1bef1da", > > > "shard" : 2, > > > "index" : "stats" > > > }, > > > "state" : "STARTED", > > > "store\_size" : "1.9g", > > > "store\_size\_in\_bytes" : 2050764870, > > > "estimated\_flushable\_memory\_size" : "877.4k", > > > "estimated\_flushable\_memory\_size\_in\_bytes" : 898541, > > > "translog\_id" : 0, > > > "translog\_operations" : 897, > > > "docs" : { > > > "num\_docs" : 1412645, > > > "max\_doc" : 1413300, > > > "deleted\_docs" : 655 > > > } > > > }, { > > > "routing" : { > > > "state" : "STARTED", > > > "primary" : true, > > > "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", > > > "relocating\_node" : null, > > > "shard" : 2, > > > "index" : "stats" > > > }, > > > "state" : "STARTED", > > > "store\_size" : "1.9g", > > > "store\_size\_in\_bytes" : 2050751585, > > > "estimated\_flushable\_memory\_size" : "876.7k", > > > "estimated\_flushable\_memory\_size\_in\_bytes" : 897826, > > > "translog\_id" : 0, > > > "translog\_operations" : 896, > > > "docs" : { > > > "num\_docs" : 1412644, > > > "max\_doc" : 1413299, > > > "deleted\_docs" : 655 > > > } > > > } ], > > > "3" : [ { > > > "routing" : { > > > "state" : "RELOCATING", > > > "primary" : true, > > > "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", > > > "relocating\_node" : "411f522e- > > > bfbc-4e0b-8a70-6451d1bef1da", > > > "shard" : 3, > > > "index" : "stats" > > > }, > > > "state" : "STARTED", > > > "store\_size" : "195m", > > > "store\_size\_in\_bytes" : 204574035, > > > "estimated\_flushable\_memory\_size" : "3.7m", > > > "estimated\_flushable\_memory\_size\_in\_bytes" : 3949835, > > > "translog\_id" : 0, > > > "translog\_operations" : 3842, > > > "docs" : { > > > "num\_docs" : 103921, > > > "max\_doc" : 103921, > > > "deleted\_docs" : 0 > > > } > > > }, { > > > "routing" : { > > > "state" : "STARTED", > > > "primary" : false, > > > "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", > > > "relocating\_node" : null, > > > "shard" : 3, > > > "index" : "stats" > > > }, > > > "state" : "STARTED", > > > "store\_size" : "195m", > > > "store\_size\_in\_bytes" : 204572396, > > > "estimated\_flushable\_memory\_size" : "3.7m", > > > "estimated\_flushable\_memory\_size\_in\_bytes" : 3949835, > > > "translog\_id" : 0, > > > "translog\_operations" : 3842, > > > "docs" : { > > > "num\_docs" : 103921, > > > "max\_doc" : 103921, > > > "deleted\_docs" : 0 > > > } > > > } ], > > > "4" : [ { > > > "routing" : { > > > "state" : "STARTED", > > > "primary" : false, > > > "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", > > > "relocating\_node" : null, > > > "shard" : 4, > > > "index" : "stats" > > > }, > > > "state" : "STARTED", > > > "store\_size" : "1.9g", > > > "store\_size\_in\_bytes" : 2106604638, > > > "estimated\_flushable\_memory\_size" : "40.9m", > > > "estimated\_flushable\_memory\_size\_in\_bytes" : 42929689, > > > "translog\_id" : 262, > > > "translog\_operations" : 43410, > > > "docs" : { > > > "num\_docs" : 1415174, > > > "max\_doc" : 1419993, > > > "deleted\_docs" : 4819 > > > } > > > }, { > > > "routing" : { > > > "state" : "STARTED", > > > "primary" : true, > > > "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", > > > "relocating\_node" : null, > > > "shard" : 4, > > > "index" : "stats" > > > }, > > > "state" : "STARTED", > > > "store\_size" : "104.9m", > > > "store\_size\_in\_bytes" : 110022541, > > > "estimated\_flushable\_memory\_size" : "3.2m", > > > "estimated\_flushable\_memory\_size\_in\_bytes" : 3392079, > > > "translog\_id" : 0, > > > "translog\_operations" : 3311, > > > "docs" : { > > > "num\_docs" : 38319, > > > "max\_doc" : 38319, > > > "deleted\_docs" : 0 > > > } > > > } ], > > > "5" : [ { > > > "routing" : { > > > "state" : "STARTED", > > > "primary" : true, > > > "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", > > > "relocating\_node" : null, > > > "shard" : 5, > > > "index" : "stats" > > > > ... > > > > read more » ---
**Author:** ![Grant\_Rodgers](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grant_rodgers/32/3204_2.png) [@Grant\_Rodgers](https://discuss.elastic.co/u/Grant_Rodgers)\ **Post date:** [July 24, 2010, 9:21pm UTC](https://discuss.elastic.co/t/issue-with-shard-replicas-being-different-in-size/3144/5 "2010-07-24T21:21:29Z")
We will reindex again when 0.9 is released; this index is not user- facing so we can afford some downtime. On Jul 24, 12:47 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Hi, > > Can you wait with the reindexing? Almost done with the new AWS specific > cloud plugin, once its done, will ping this mailing list with docs on how to > use it. For now, decided to go with the formal AWS SDK to simplify things, > though in the future might implement a proper non blocking myself. > > -shay.banon > > On Sat, Jul 24, 2010 at 10:32 PM, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com) wrote: > > > Yes, actually, after the index we also replaced two of the nodes. The > > procedure we used was to stop one node, wait for the shards to > > stabilize on the other two, then start the new one, and wait for the > > shards to stabilize again. We did this twice, so there was a lot of > > relocating going on. Unfortunately I didn't test the numbers before > > the node swaps, but we are doing a full reindex this weekend so we > > will see if the issue comes up again without them. We are using 0.8.0 > > and the s3 gateway. > > > We're looking forward to 0.9 quite keenly. The improved shard > > management mentioned here, plus work directory reuse, new cloud > > gateway, and new facets are all things we can use! Any idea of a > > timetable for the release? > > > Thanks, > > Grant > > > On Jul 24, 2:56 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > Hi, > > > > ``` > > > The problem of the primary, when shutdown, it gets allocated and > > > > > > ``` > > > > > > restored from the gateway (instead of having one of the replicas become > > > the > > > primary) is fixed in 0.9. The difference between the two is strange > > > (replica > > > and primary). I have fixed in 0.9 some cases where this might happen > > > during > > > relocation of shards, did that happen to you? Also, are you using the s3 > > > gateway? > > > > -shay.banon > > > > On Sat, Jul 24, 2010 at 12:32 AM, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com) > > > wrote: > > > > > > > Yesterday I did a full reindex of an index with ~8 million documents. > > > > I had 3 nodes on EC2 large instances. There were 8 clients indexing > > > > documents at the rate of ~1100 documents per minute per client. > > > > > Today I did some match\_all count queries to verify the index size, and > > > > I notice that every other result is different: sometimes 5m, sometimes > > > > 8m. I look at the index status and I see that two shards have very > > > > differently sized replicas: The primaries have \<120k documents, and > > > > the replicas have \>1.3m documents. > > > > > I tried stopping nodes to see if I could get the correct replicas to > > > > become the primaries, but it seems they may be stored in the gateway > > > > as the much smaller size rather than the correct size. It appears the > > > > behavior is that if a replica goes down, it is restored from the > > > > primary, but if a primary goes down, it is restored from the gateway. > > > > > Here is the output from the index status command. Notice that shard 4 > > > > has a big difference between the size of the replica and the primary. > > > > Shard 3 originally had the same issue, but the correct replica was > > > > lost when the node with that replica was accidentally restarted > > > > (oops!) > > > > > { > > > > "ok" : true, > > > > "\_shards" : { > > > > "total" : 12, > > > > "successful" : 12, > > > > "failed" : 0 > > > > }, > > > > "indices" : { > > > > "stats" : { > > > > "aliases" : , > > > > "settings" : { > > > > "index.number\_of\_shards" : "6", > > > > "index.number\_of\_replicas" : "1" > > > > }, > > > > "store\_size" : "20.3g", > > > > "store\_size\_in\_bytes" : 21814342869, > > > > "estimated\_flushable\_memory\_size" : "96.5m", > > > > "estimated\_flushable\_memory\_size\_in\_bytes" : 101234949, > > > > "translog\_operations" : 102062, > > > > "docs" : { > > > > "num\_docs" : 5738311, > > > > "max\_doc" : 5744803, > > > > "deleted\_docs" : 6492 > > > > }, > > > > "shards" : { > > > > "0" : [ { > > > > "routing" : { > > > > "state" : "STARTED", > > > > "primary" : false, > > > > "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", > > > > "relocating\_node" : null, > > > > "shard" : 0, > > > > "index" : "stats" > > > > }, > > > > "state" : "STARTED", > > > > "store\_size" : "2.6g", > > > > "store\_size\_in\_bytes" : 2842186930, > > > > "estimated\_flushable\_memory\_size" : "298.8k", > > > > "estimated\_flushable\_memory\_size\_in\_bytes" : 306021, > > > > "translog\_id" : 0, > > > > "translog\_operations" : 306, > > > > "docs" : { > > > > "num\_docs" : 1352920, > > > > "max\_doc" : 1352920, > > > > "deleted\_docs" : 0 > > > > } > > > > }, { > > > > "routing" : { > > > > "state" : "RELOCATING", > > > > "primary" : true, > > > > "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", > > > > "relocating\_node" : "411f522e- > > > > bfbc-4e0b-8a70-6451d1bef1da", > > > > "shard" : 0, > > > > "index" : "stats" > > > > }, > > > > "state" : "STARTED", > > > > "store\_size" : "2.6g", > > > > "store\_size\_in\_bytes" : 2842196644, > > > > "estimated\_flushable\_memory\_size" : "298.8k", > > > > "estimated\_flushable\_memory\_size\_in\_bytes" : 306021, > > > > "translog\_id" : 1, > > > > "translog\_operations" : 306, > > > > "docs" : { > > > > "num\_docs" : 1352920, > > > > "max\_doc" : 1352920, > > > > "deleted\_docs" : 0 > > > > } > > > > } ], > > > > "1" : [ { > > > > "routing" : { > > > > "state" : "STARTED", > > > > "primary" : false, > > > > "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", > > > > "relocating\_node" : null, > > > > "shard" : 1, > > > > "index" : "stats" > > > > }, > > > > "state" : "STARTED", > > > > "store\_size" : "1.9g", > > > > "store\_size\_in\_bytes" : 2109353140, > > > > "estimated\_flushable\_memory\_size" : "304.8k", > > > > "estimated\_flushable\_memory\_size\_in\_bytes" : 312123, > > > > "translog\_id" : 0, > > > > "translog\_operations" : 323, > > > > "docs" : { > > > > "num\_docs" : 1413888, > > > > "max\_doc" : 1418829, > > > > "deleted\_docs" : 4941 > > > > } > > > > }, { > > > > "routing" : { > > > > "state" : "RELOCATING", > > > > "primary" : true, > > > > "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", > > > > "relocating\_node" : "411f522e- > > > > bfbc-4e0b-8a70-6451d1bef1da", > > > > "shard" : 1, > > > > "index" : "stats" > > > > }, > > > > "state" : "STARTED", > > > > "store\_size" : "1.9g", > > > > "store\_size\_in\_bytes" : 2109369148, > > > > "estimated\_flushable\_memory\_size" : "304.8k", > > > > "estimated\_flushable\_memory\_size\_in\_bytes" : 312123, > > > > "translog\_id" : 1, > > > > "translog\_operations" : 323, > > > > "docs" : { > > > > "num\_docs" : 1413888, > > > > "max\_doc" : 1418829, > > > > "deleted\_docs" : 4941 > > > > } > > > > } ], > > > > "2" : [ { > > > > "routing" : { > > > > "state" : "RELOCATING", > > > > "primary" : false, > > > > "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", > > > > "relocating\_node" : "411f522e- > > > > bfbc-4e0b-8a70-6451d1bef1da", > > > > "shard" : 2, > > > > "index" : "stats" > > > > }, > > > > "state" : "STARTED", > > > > "store\_size" : "1.9g", > > > > "store\_size\_in\_bytes" : 2050764870, > > > > "estimated\_flushable\_memory\_size" : "877.4k", > > > > "estimated\_flushable\_memory\_size\_in\_bytes" : 898541, > > > > "translog\_id" : 0, > > > > "translog\_operations" : 897, > > > > "docs" : { > > > > "num\_docs" : 1412645, > > > > "max\_doc" : 1413300, > > > > "deleted\_docs" : 655 > > > > } > > > > }, { > > > > "routing" : { > > > > "state" : "STARTED", > > > > "primary" : true, > > > > "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", > > > > "relocating\_node" : null, > > > > "shard" : 2, > > > > "index" : "stats" > > > > }, > > > > "state" : "STARTED", > > > > "store\_size" : "1.9g", > > > > "store\_size\_in\_bytes" : 2050751585, > > > > "estimated\_flushable\_memory\_size" : "876.7k", > > > > "estimated\_flushable\_memory\_size\_in\_bytes" : 897826, > > > > "translog\_id" : 0, > > > > "translog\_operations" : 896, > > > > "docs" : { > > > > "num\_docs" : 1412644, > > > > "max\_doc" : 1413299, > > > > "deleted\_docs" : 655 > > > > } > > > > } ], > > > > "3" : [ { > > > > "routing" : { > > > > "state" : "RELOCATING", > > > > "primary" : true, > > > > "node" : "9a54c6aa-469d-46ff-a574-dc24c969a0b0", > > > > "relocating\_node" : "411f522e- > > > > bfbc-4e0b-8a70-6451d1bef1da", > > > > "shard" : 3, > > > > "index" : "stats" > > > > }, > > > > "state" : "STARTED", > > > > "store\_size" : "195m", > > > > "store\_size\_in\_bytes" : 204574035, > > > > "estimated\_flushable\_memory\_size" : "3.7m", > > > > "estimated\_flushable\_memory\_size\_in\_bytes" : 3949835, > > > > "translog\_id" : 0, > > > > "translog\_operations" : 3842, > > > > "docs" : { > > > > "num\_docs" : 103921, > > > > "max\_doc" : 103921, > > > > "deleted\_docs" : 0 > > > > } > > > > }, { > > > > "routing" : { > > > > "state" : "STARTED", > > > > "primary" : false, > > > > "node" : "ccc84aea-aabb-40e3-837c-af1bcb1b764a", > > > > "relocating\_node" : null, > > > > "shard" : 3, > > > > "index" : "stats" > > > > }, > > > > "state" : "STARTED", > > > > "store\_size" : > > ... > > read more » ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:21am UTC](https://discuss.elastic.co/t/issue-with-shard-replicas-being-different-in-size/3144/6 "2017-07-06T04:21:42Z")