Elasticsearch Index Split

tusharnemade · October 1, 2021, 5:32am

Hello Team

I am using Elasticsearch version 7.8.0

I used API "SPLIT" to break my big index into multiple shards.

After split was concluded successfully....

I found that my new SPLITTED Index [ myindex-000797-sp ] is double the size of Source Index ...

health status index               uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   myindex-000797    aHevVd8iSZyy4Wmfg1Q4rw   5   1   1057316946            0      1.5tb        798.9gb
green  open   myindex-000797-sp FUU40j43SiKNkDk1Bprbcg  10   1   1057316946    829935320      3.1tb          1.6tb

Could you please help me to understand why is so ... and how should i reduce the SIZE of new SPLITTED Index.

Thanks

warkolm · October 1, 2021, 5:41am

Try running a force merge on the new index, you need to purge the deleted documents.

Also, 7.8 is nearly EOL, please consider upgrading

tusharnemade · October 4, 2021, 3:41am

Hello @warkolm

Thank You for your suggestion .. It was helped me ..

health status index                        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   myindex-000797-sp            FUU40j43SiKNkDk1Bprbcg  10   1 1057316946            0      1.4tb        533.5gb

By this we can conclude that , after splitting the index , we must run force merge to release the unwanted occupied space.

Regarding Upgrading Elasticsearch from version 7.8.0 to new one will take some time ... as many other API's are related to this cluster.
We will do it .. but not very soon..

Thank You for your response.

tusharnemade · October 4, 2021, 3:45am

Now my second question is :

If you see REPLICA sizes ...

index             shard prirep state        docs   store ip           node
myindex-000797-sp 3     r      STARTED 104626098 158.2gb 10.10.10.250 ESN-7
myindex-000797-sp 3     p      STARTED 104626098  52.7gb 10.10.10.248 ESN-5
myindex-000797-sp 5     r      STARTED 104627552 159.9gb 10.10.10.247 ESN-4
myindex-000797-sp 5     p      STARTED 104627552  52.9gb 10.10.10.244 ESN-1
myindex-000797-sp 2     p      STARTED 104625084  52.7gb 10.10.10.248 ESN-5
myindex-000797-sp 2     r      STARTED 104625084  52.7gb 10.10.10.244 ESN-1
myindex-000797-sp 7     p      STARTED 104620908  52.7gb 10.10.10.246 ESN-3
myindex-000797-sp 7     r      STARTED 104620908   160gb 10.10.10.247 ESN-4
myindex-000797-sp 9     p      STARTED 107395688  54.1gb 10.10.10.245 ESN-2
myindex-000797-sp 9     r      STARTED 107395688  54.1gb 10.10.10.247 ESN-4
myindex-000797-sp 6     p      STARTED 107386745  54.1gb 10.10.10.246 ESN-3
myindex-000797-sp 6     r      STARTED 107386745  54.1gb 10.10.10.249 ESN-6
myindex-000797-sp 1     r      STARTED 107388915  54.1gb 10.10.10.246 ESN-3
myindex-000797-sp 1     p      STARTED 107388915  54.1gb 10.10.10.250 ESN-7
myindex-000797-sp 4     r      STARTED 107382031 159.9gb 10.10.10.249 ESN-6
myindex-000797-sp 4     p      STARTED 107382031  54.1gb 10.10.10.244 ESN-1
myindex-000797-sp 8     p      STARTED 104643368  52.9gb 10.10.10.245 ESN-2
myindex-000797-sp 8     r      STARTED 104643368  52.9gb 10.10.10.249 ESN-6
myindex-000797-sp 0     p      STARTED 104620557  52.7gb 10.10.10.250 ESN-7
myindex-000797-sp 0     r      STARTED 104620557  52.8gb 10.10.10.248 ESN-5

Its still has some of them as 3x times of PRIMARY shard.

How to fix this ?

tusharnemade · October 4, 2021, 6:33am

Now After Reallocating Them to another nodes of same cluster ... has free the space from REPLICA's ..

index             shard prirep state        docs  store ip           node
myindex-000797-sp 0     p      STARTED 104620557 52.7gb 10.10.10.254 ESNW-4
myindex-000797-sp 0     r      STARTED 104620557 52.7gb 10.10.10.11  ESNW-13
myindex-000797-sp 1     r      STARTED 107388915 54.1gb 10.10.10.252 ESNW-2
myindex-000797-sp 1     p      STARTED 107388915 54.1gb 10.10.10.4   ESNW-6
myindex-000797-sp 2     r      STARTED 104625084 52.7gb 10.10.10.11  ESNW-13
myindex-000797-sp 2     p      STARTED 104625084 52.7gb 10.10.10.5   ESNW-7
myindex-000797-sp 3     p      STARTED 104626098 52.7gb 10.10.10.252 ESNW-2
myindex-000797-sp 3     r      STARTED 104626098 52.7gb 10.10.10.9   ESNW-11
myindex-000797-sp 4     r      STARTED 107382031 54.1gb 10.10.10.251 ESNW-1
myindex-000797-sp 4     p      STARTED 107382031 54.1gb 10.10.10.253 ESNW-3
myindex-000797-sp 5     p      STARTED 104627552 52.9gb 10.10.10.7   ESNW-9
myindex-000797-sp 5     r      STARTED 104627552 52.9gb 10.10.10.5   ESNW-7
myindex-000797-sp 6     p      STARTED 107386745 54.1gb 10.10.10.6   ESNW-8
myindex-000797-sp 6     r      STARTED 107386745 54.1gb 10.10.10.8   ESNW-10
myindex-000797-sp 7     r      STARTED 104620908 52.7gb 10.10.10.8   ESNW-10
myindex-000797-sp 7     p      STARTED 104620908 52.7gb 10.10.10.10  ESNW-12
myindex-000797-sp 8     r      STARTED 104643368 52.9gb 10.10.10.13  ESNW-15
myindex-000797-sp 8     p      STARTED 104643368 52.9gb 10.10.10.12  ESNW-14
myindex-000797-sp 9     r      STARTED 107395688 54.1gb 10.10.10.6   ESNW-8
myindex-000797-sp 9     p      STARTED 107395688 54.1gb 10.10.10.3   ESNW-5

system · November 1, 2021, 6:34am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.