We run cluster on 5.6.3 , a 45 node cluster with single replica and shard size is around 70 GB . On some occasions shard size is expanding more than 75% of usual size and reaching to 120GB . This is causing diskspace high water mark issues.
- shard size expansion/contraction by 75% of size, is this normal for a healthy cluster like 70 GB -> 120 GB -> 75GB ??
- ANy specific settings we can enable so size doesn;t differ with this marginal difference
Inde shard Replica/primary STATUS Normal size in GB Expanded size in GB
inde-name 0 p STARTED 70.9 116.3
inde-name 0 r STARTED 75.4 72.4
inde-name 1 p STARTED 64.5 106.6
inde-name 1 r STARTED 76 74.4
inde-name 2 p STARTED 84 104.4
inde-name 2 r STARTED 62.6 80
inde-name 3 p STARTED 64.4 117
inde-name 3 r STARTED 79.8 75.4
inde-name 4 p STARTED 74.3 99.6
inde-name 4 r STARTED 58.3 87.2
inde-name 5 r STARTED 68.1 97.5
inde-name 5 p STARTED 68.1 92.4
inde-name 6 p STARTED 60.6 100.3
inde-name 6 r STARTED 80.4 72.2
inde-name 7 p STARTED 59.1 107.9
inde-name 7 r STARTED 69.1 70.7
inde-name 8 p STARTED 61.6 98.3
inde-name 8 r STARTED 81.1 69.4
inde-name 9 p STARTED 74.1 112.5
inde-name 9 r STARTED 62.2 84.3
inde-name 10 p STARTED 65.9 114.3
inde-name 10 r STARTED 77.3 76.3
inde-name 11 r STARTED 66.9 101.8
inde-name 11 p STARTED 68.4 80.6
inde-name 12 p STARTED 63.7 107.9
inde-name 12 r STARTED 77.8 70
inde-name 13 r STARTED 65.3 110.8
inde-name 13 p STARTED 80.4 80
inde-name 14 r STARTED 65.6 101
inde-name 14 p STARTED 84.3 76.5
inde-name 15 p STARTED 69.3 110.6
inde-name 15 r STARTED 57.8 73
inde-name 16 r STARTED 67.3 90.4
inde-name 16 p STARTED 68.5 83.7
inde-name 17 p STARTED 61.2 87.4
inde-name 17 r STARTED 69.6 83
inde-name 18 r STARTED 56.3 105
inde-name 18 p STARTED 74.1 62.7
inde-name 19 p STARTED 62.2 103.8
inde-name 19 r STARTED 78.1 81.5
inde-name 20 r STARTED 70.7 95.7
inde-name 20 p STARTED 57.9 86.9
inde-name 21 r STARTED 61.4 110.9
inde-name 21 p STARTED 74.4 73.5
inde-name 22 r STARTED 57.5 97.2
inde-name 22 p STARTED 69 92
inde-name 23 p STARTED 62 106.3
inde-name 23 r STARTED 81.8 77.1
inde-name 24 r STARTED 66.2 112.2
inde-name 24 p STARTED 71.6 69.5
inde-name 25 r STARTED 65.5 108.5
inde-name 25 p STARTED 68.1 73
inde-name 26 r STARTED 63.4 104.2
inde-name 26 p STARTED 56.9 70.6
inde-name 27 r STARTED 60.1 104.1
inde-name 27 p STARTED 78.9 75.1
inde-name 28 r STARTED 62.9 104.5
inde-name 28 p STARTED 66 87.6
inde-name 29 p STARTED 64.4 111
inde-name 29 r STARTED 77.4 76.8
inde-name 30 r STARTED 71.3 110.8
inde-name 30 p STARTED 70.3 75.1
inde-name 31 p STARTED 55.9 89.7
inde-name 31 r STARTED 66.5 83.9
inde-name 32 p STARTED 64 94.5
inde-name 32 r STARTED 68.8 85.1
inde-name 33 r STARTED 71 109.3
inde-name 33 p STARTED 80 71.2
inde-name 34 p STARTED 71 109
inde-name 34 r STARTED 70 77.3
inde-name 35 r STARTED 72 108.1
inde-name 35 p STARTED 65.8 80.2
inde-name 36 p STARTED 71.7 104.3
inde-name 36 r STARTED 63.3 72.2
inde-name 37 p STARTED 64.6 109.6
inde-name 37 r STARTED 68.1 65.3
inde-name 38 r STARTED 57.7 109.5
inde-name 38 p STARTED 89.3 83
inde-name 39 p STARTED 67.8 100.9
inde-name 39 r STARTED 70.8 65.6
inde-name 40 r STARTED 74.5 101.5
inde-name 40 p STARTED 65.8 70.1
inde-name 41 r STARTED 61.4 92.7
inde-name 41 p STARTED 78.8 80.9
inde-name 42 r STARTED 61 99.7
inde-name 42 p STARTED 72.2 78
inde-name 43 r STARTED 70 114
inde-name 43 p STARTED 73.4 70
inde-name 44 p STARTED 64 110.6
inde-name 44 r STARTED 77.2 79.5