Scaling ElasticSearch for many indexes

[JanGoAutonomous (Jan Meznik)]

TL;DR: How can I remediate running out of shards when using many indexes?

---

My application creates separate Elasticsearch indexes for each user of the app.
This separation ensures that each mapping can be different, and the user-data is isolated as to not "leak" over.

Each index uses about 2 shards and 1 replica. Unfortunately, with the growing number of users, I am running out of shards with errors such as:

 this action would add [2] shards, but this cluster currently has [999]/[1000] maximum normal shards open

It seems like increasing the number of shards is discouraged.

Am I using Elasticsearch wrong? How can I change my application to support that each user has its own "search" index?

[Christian_Dahlqvist]

Elasticsearch in my experience scales badly with large number of small indices like you get when you have indices per user. This has been improved in recent versions, but there is still a reason the default limit is there.

This has been asked a number of times over the years (most of the refrences I could find are a bit old) and I believe the recommendation still is to avoid trying to scale this way. If this in any way has changed I am sure someone will chime in and correct me.

Effective separation of tenant data in latest release of ElasticSearch

Hi All, We want to use ElasticSearch as a multi-tenant store , each tenant would have different requirement for document type/schema. What is the best way to store data wrt cost, manageability in this regard ? 1> Each tenant having separate index with varying document types may not be efficient? 2> A set of tenants may fall into one index with varying document types but With ElasticSearch's removal of mapping types mentioned in link It seems to be possible only through have custom type as…

Sizing and configuration for multi-tenant application

Hi, I've been using ES in production for a long time for our application, but it is currently single tenant, and so I have dedicated ES instances running for each client. We are moving towards a multi-tenant platform and I just want to make sure I'm approaching this well from a hardware/config perspective. Currently, I'm storing a total of roughly 7TB of data across all these tenants. The plan is to move this all into a multi-tenant system. This could all be in one huge cluster, or we could se…

@JanGoAutonomous

How can I change my application to support that each user has its own "search" index?

Assuming the indices are small I would start by setting the number of primary shards to 1. That will reduce the shard count by 50%.

The best way forward after that depends a bit on how many users you will need to support and the use case in terms of mappings and queries/aggregations used. Another aspect is whether users have direct access to Elasticsearch or not. If you can provide some additional deatils on this we may be able to provide some suggestions.

[JanGoAutonomous (Jan Meznik)]

In the foreseeable future, I'll have 100 users (and even more in the future), each with around 5 indexes. Each index ranges from 10mb - 1Gb.

I am considering making only 1 index per user and combine the different types of documents there. However, given that I occasionally need to update the mappings (analyzers, tokenizers, fields), I think this will add much more complexity to the maintenance.