Dynamic Shards (Incremental Resharding)

Elastic Stack Elasticsearch
1

Hello everyone,

One of the most common challenges with big data is dealing with increasing volume of data. The traditional solution in ElasticSearch is dynamically creating new indices - which means that the latest documents are found in the new created indices. This allows implementing Hot/Warm Architecture with built-in mechanism like rollover index and index lifecycle management.

I wounder: Instead of dynamically creating new indices, is there a way of dynamically creating new shards? This approach is supported in Solr: there is an API for creating new empty shards in existing index. Is there any way to do so in ElasticSearch? I haven't seen any API allowing creating new shards dynamically as solr implemented. Only noticed shrinking index to lower shards or splitting index to more shards - but not an option to create a new empty shard.

The motivation in dynamic shards instead if dynamic indices - is that ranking between documents in the same index is much more accurate than ranking documents between different indices (distributed TF/IDF).

2

Check out the [https://www.elastic.co/guide/en/elasticsearch/reference/7.1/indices-rollover-index.html](rollover index API)

3

Thanks for your response, but rollover API allows creating new indices dynamically (and exposing them with an appropriate alias). It doesn't change any of the shards of the current indices.

4

There is no way to dynamically create new shards apart from splitting the index. Using time-based indices is the normal way of getting around this.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.