Increase the number of shrads

[cool_arun]

How to increase the the number of shrads of elasticsearch index

[dadoonet (David Pilato)]

For a new index?

PUT /my_temp_index
{
    "settings": {
        "number_of_shards" :   10
    }
}

For an existing index, it might be possible to call: https://www.elastic.co/guide/en/elasticsearch/reference/6.3/indices-split-index.html

Otherwise you need to reindex. Use the Reindex API for that: https://www.elastic.co/guide/en/elasticsearch/reference/6.3/docs-reindex.html

[cool_arun]

Thank you so much for the quick reply, i have 100 million documents in the index and the number of shrads are 5 and i want to increase that to 30, so my questions is which would be the fast and efficient way to achieve this.

[Christian_Dahlqvist]

As you may not have created the index with settings that allow you to use the split index API, the only way will be to reindex your data into a new index with the correct settings.

[mikebd (Michael Ben-David)]

Do you really want to have 30 shards? Usual guidance is for a shard to be 30 GB before considering multiple shards, ymmv depending on resources and usage patterns.

It is more common to add replicas to increase distributed search capability depending on resources available.

[cool_arun]

thanks,
no i didn't do the setting for the split index API. is it a default setting.

[Christian_Dahlqvist]

No it is not, so you will need to reindex.

[cool_arun]

thanks christian, so if i go for re-indexing for the 100 million documents, will it be a slow process or it depends on my disk space. the reason is i want to know approximate time it will take for re-indexing, is there any way can i calculate that.

[Christian_Dahlqvist]

This depends on your data, mappings and hardware so there is no general way to calculate how long it will take.

[Abhilash_Bolla]

If you aren't using aggressive analyzers like ngram, stemmers, synonyms, etc and if number of fields are < 15 per document, it should be over in half a day. Are you performing the reindex in a cluster or a single node? If you do it in a cluster, beware of the client timeouts. For such large data of 100 million documents, I would prefer to write a custom script than using a single HTTP request (like what reindex API does).