HI, my ES data can grow up to 270,000 indexes. I'm creating one index per user. Is there a ES limit to create indexes? Each index has one shard and one replica only. My cluster have 3 master nodes and 2 data nodes
We recommend no more than 600 shards per node.
Why do you need one per user? What sort of data is it?
Elasticsearch does not impose any strict limit to the number of indices or shards, but that does not mean that there are not practical limits. Having an index per user adds a lot of flexibility and isolation, but unfortunately does not scale well at all. The mappings of each index together with information about the location of the shards need to be kept in the cluster state, which grows quickly the more indices and shards you add. As it grows changes to the cluster state, e.g. shard relocations and mapping changes, will take longer and longer to apply, until the cluster no longer works very well.
Hi Mark.
My document type is a question list. My user is a student. Each student has many questions list. Until the end of the year, the user is expected to have 4,000 questions list.
Another strategy that I've think is create one single index and 10 shards using _routing field to index document ( https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-routing-field.html ). What you think?
How much data do you have in total?
Now I have 82,000,000 of documents (20Gb). I expect to double it until december
That easily fits in a single index. If you pick 5-10 primary shards you will probably be fine for the foreseeable future.
1 Like
thanks a lot
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.