Dynamic Shards (Incremental Resharding)

Hello everyone,

One of the most common challenges with big data is dealing with increasing volume of data. The traditional solution in ElasticSearch is dynamically creating new indices - which means that the latest documents are found in the new created indices. This allows implementing Hot/Warm Architecture with built-in mechanism like rollover index and index lifecycle management.

I wounder: Instead of dynamically creating new indices, is there a way of dynamically creating new shards? This approach is supported in Solr: there is an API for creating new empty shards in existing index. Is there any way to do so in ElasticSearch? I haven't seen any API allowing creating new shards dynamically as solr implemented. Only noticed shrinking index to lower shards or splitting index to more shards - but not an option to create a new empty shard.

The motivation in dynamic shards instead if dynamic indices - is that ranking between documents in the same index is much more accurate than ranking documents between different indices (distributed TF/IDF).

Check out the [https://www.elastic.co/guide/en/elasticsearch/reference/7.1/indices-rollover-index.html](rollover index API)

Thanks for your response, but rollover API allows creating new indices dynamically (and exposing them with an appropriate alias). It doesn't change any of the shards of the current indices.

There is no way to dynamically create new shards apart from splitting the index. Using time-based indices is the normal way of getting around this.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.