Scaling ElasticSearch for many indexes

Elastic Stack Elasticsearch
1

TL;DR: How can I remediate running out of shards when using many indexes?

My application creates separate Elasticsearch indexes for each user of the app.
This separation ensures that each mapping can be different, and the user-data is isolated as to not "leak" over.

Each index uses about 2 shards and 1 replica. Unfortunately, with the growing number of users, I am running out of shards with errors such as:

 this action would add [2] shards, but this cluster currently has [999]/[1000] maximum normal shards open

It seems like increasing the number of shards is discouraged.

Am I using Elasticsearch wrong? How can I change my application to support that each user has its own "search" index?

2

Elasticsearch in my experience scales badly with large number of small indices like you get when you have indices per user. This has been improved in recent versions, but there is still a reason the default limit is there.

This has been asked a number of times over the years (most of the refrences I could find are a bit old) and I believe the recommendation still is to avoid trying to scale this way. If this in any way has changed I am sure someone will chime in and correct me.

Assuming the indices are small I would start by setting the number of primary shards to 1. That will reduce the shard count by 50%.

The best way forward after that depends a bit on how many users you will need to support and the use case in terms of mappings and queries/aggregations used. Another aspect is whether users have direct access to Elasticsearch or not. If you can provide some additional deatils on this we may be able to provide some suggestions.

3

In the foreseeable future, I'll have 100 users (and even more in the future), each with around 5 indexes. Each index ranges from 10mb - 1Gb.

I am considering making only 1 index per user and combine the different types of documents there. However, given that I occasionally need to update the mappings (analyzers, tokenizers, fields), I think this will add much more complexity to the maintenance.