# How to scale an ES deployment to millions of tenants with different data schemas
**URL:** https://discuss.elastic.co/t/how-to-scale-an-es-deployment-to-millions-of-tenants-with-different-data-schemas/19844
**Category:** Elasticsearch
**Created:** [September 17, 2014, 1:05pm UTC](https://discuss.elastic.co/t/how-to-scale-an-es-deployment-to-millions-of-tenants-with-different-data-schemas/19844 "2014-09-17T13:05:01Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@zivs](https://discuss.elastic.co/u/zivs)
#### Post date: [September 17, 2014, 1:05pm UTC](https://discuss.elastic.co/t/how-to-scale-an-es-deployment-to-millions-of-tenants-with-different-data-schemas/19844/1 "2014-09-17T13:05:01Z")
Hi,
we are considering using ES as a primary data-source for a new project.
our data is generated by millions of different users, _each having a
relatively small number of documents, yet each having a different data
schema._
_we are considering several approaches:_
- index per user - we are concerned with scaling the ES cluster to support
millions of indexes, each having relatively small number of docs.
- all users colocated on a single index - we are concerned that an ES index
will not support millions of different fields (as each user has a different
data schema).
- mix of the two above - having X users colocated on a single index, and
having Y such indexes to host our entire user population.
- implementing some kind of a "mapping layer" that maps users' schema onto
generic fields in one or more indexes.
this would probably work, but of course is harder to implement & maintain.
_so my questions:_
- are there production deployments out there that have a million active
indexes? what do they look like?
- how many different fields does it make sense to host in a single index?
would it scale to millions of fields in a single index?
- are there other ways to go about this that we have overlooked?
thanks!!
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f98651e6-5e6a-4ed8-aba8-b5e91078f036%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f98651e6-5e6a-4ed8-aba8-b5e91078f036%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)
#### Post date: [September 17, 2014, 1:21pm UTC](https://discuss.elastic.co/t/how-to-scale-an-es-deployment-to-millions-of-tenants-with-different-data-schemas/19844/2 "2014-09-17T13:21:09Z")
First, you should really read this:
[http://aphyr.com/posts/317-call-me-maybe-elasticsearch](http://aphyr.com/posts/317-call-me-maybe-elasticsearch) regarding using ES
as a single source of truth
Millions of indexes is not advisable, unless you plan on having millions of
servers. Depending on index size and write frequency to them, you don't
want to have more than a few dozen indexes per machine (including
replicas). This is because of concerns of memory, CPU, I/O and file
descriptors.
One big single index may present its own problems due to the different
schemas, although it may be solvable using dynamic index templates
[http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/custom-dynamic-mapping.html#dynamic-templates](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/custom-dynamic-mapping.html#dynamic-templates).
I will still expect you to have issues with number of shards (basically,
running out of shards at some point).
Therefore I will try and find a middle way here, using probably some sort
of a mapping mechanism. Even also time based if its applicable.
Re your questions:
- are there production deployments out there that have a million active
indexes? what do they look like?
I'm not aware of such
- how many different fields does it make sense to host in a single index?
would it scale to millions of fields in a single index?
You mean in a single document. I recall seeing Shay suggesting not to go
over the 100 threshold or so. Lucene really isn't optimized for scaling
vertically, especially in the document level.
- are there other ways to go about this that we have overlooked?
Maybe look at your data model and try to re-arrange it.
--
Itamar Syn-Hershko
[http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
Freelance Developer & Consultant
Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/)
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsNarKqML9q4E%3DvdrDp80mC\_rS5VXfJWkT2%3D7WghzwACg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsNarKqML9q4E%3DvdrDp80mC_rS5VXfJWkT2%3D7WghzwACg%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@zivs](https://discuss.elastic.co/u/zivs)
#### Post date: [September 17, 2014, 1:35pm UTC](https://discuss.elastic.co/t/how-to-scale-an-es-deployment-to-millions-of-tenants-with-different-data-schemas/19844/3 "2014-09-17T13:35:47Z")
thanks for the prompt reply!
one thing though - when using a single multi-tenant index, my concerns are
not around the number of fields per doc (which is small, less than 50),
but rather the fact that since each tenant has different fields, the
accumulated number of fields in such an index will be huge.
i.e. tenant 1 has fields F11..F1n, tenant 2 has fields F21..F2n, ...
these fields are distinct so the number of fields for the multi-tenant
index will grow to millions quickly.
will such an indexing methodology work in ES?
thanks!
On Wednesday, September 17, 2014 4:21:17 PM UTC+3, Itamar Syn-Hershko wrote:
> First, you should really read this:
> [Jepsen: Elasticsearch](http://aphyr.com/posts/317-call-me-maybe-elasticsearch) regarding using ES
> as a single source of truth
>
> Millions of indexes is not advisable, unless you plan on having millions
> of servers. Depending on index size and write frequency to them, you don't
> want to have more than a few dozen indexes per machine (including
> replicas). This is because of concerns of memory, CPU, I/O and file
> descriptors.
>
> One big single index may present its own problems due to the different
> schemas, although it may be solvable using dynamic index templates
> [http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/custom-dynamic-mapping.html#dynamic-templates](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/custom-dynamic-mapping.html#dynamic-templates).
> I will still expect you to have issues with number of shards (basically,
> running out of shards at some point).
>
> Therefore I will try and find a middle way here, using probably some sort
> of a mapping mechanism. Even also time based if its applicable.
>
> Re your questions:
>
> - are there production deployments out there that have a million active
> indexes? what do they look like?
>
> I'm not aware of such
>
> - how many different fields does it make sense to host in a single index?
> would it scale to millions of fields in a single index?
>
> You mean in a single document. I recall seeing Shay suggesting not to go
> over the 100 threshold or so. Lucene really isn't optimized for scaling
> vertically, especially in the document level.
>
> - are there other ways to go about this that we have overlooked?
>
> Maybe look at your data model and try to re-arrange it.
>
> --
>
> Itamar Syn-Hershko
> [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
> Freelance Developer & Consultant
> Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/)
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/23a8484a-dcfc-4c8a-bc9d-a02bc4280985%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/23a8484a-dcfc-4c8a-bc9d-a02bc4280985%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)
#### Post date: [September 17, 2014, 1:38pm UTC](https://discuss.elastic.co/t/how-to-scale-an-es-deployment-to-millions-of-tenants-with-different-data-schemas/19844/4 "2014-09-17T13:38:38Z")
This will still mean less overhead than having those distinct field in
discreet indexes. I wouldn't worry about that.
--
Itamar Syn-Hershko
[http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
Freelance Developer & Consultant
Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/)
On Wed, Sep 17, 2014 at 4:35 PM, Ziv Shalev [zivs@wix.com](mailto:zivs@wix.com) wrote:
> thanks for the prompt reply!
> one thing though - when using a single multi-tenant index, my concerns are
> not around the number of fields per doc (which is small, less than 50),
> but rather the fact that since each tenant has different fields, the
> accumulated number of fields in such an index will be huge.
>
> i.e. tenant 1 has fields F11..F1n, tenant 2 has fields F21..F2n, ...
> these fields are distinct so the number of fields for the multi-tenant
> index will grow to millions quickly.
>
> will such an indexing methodology work in ES?
>
> thanks!
>
> On Wednesday, September 17, 2014 4:21:17 PM UTC+3, Itamar Syn-Hershko
> wrote:
>
> > First, you should really read this: [Aphyr - Posts](http://aphyr.com/posts/)
> > 317-call-me-maybe-elasticsearch regarding using ES as a single source of
> > truth
> >
> > Millions of indexes is not advisable, unless you plan on having millions
> > of servers. Depending on index size and write frequency to them, you don't
> > want to have more than a few dozen indexes per machine (including
> > replicas). This is because of concerns of memory, CPU, I/O and file
> > descriptors.
> >
> > One big single index may present its own problems due to the different
> > schemas, although it may be solvable using dynamic index templates
> > [http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/custom-dynamic-mapping.html#dynamic-templates](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/custom-dynamic-mapping.html#dynamic-templates).
> > I will still expect you to have issues with number of shards (basically,
> > running out of shards at some point).
> >
> > Therefore I will try and find a middle way here, using probably some sort
> > of a mapping mechanism. Even also time based if its applicable.
> >
> > Re your questions:
> >
> > - are there production deployments out there that have a million active
> > indexes? what do they look like?
> >
> > I'm not aware of such
> >
> > - how many different fields does it make sense to host in a single index?
> > would it scale to millions of fields in a single index?
> >
> > You mean in a single document. I recall seeing Shay suggesting not to go
> > over the 100 threshold or so. Lucene really isn't optimized for scaling
> > vertically, especially in the document level.
> >
> > - are there other ways to go about this that we have overlooked?
> >
> > Maybe look at your data model and try to re-arrange it.
> >
> > --
> >
> > Itamar Syn-Hershko
> > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
> > Freelance Developer & Consultant
> > Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/)
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/23a8484a-dcfc-4c8a-bc9d-a02bc4280985%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/23a8484a-dcfc-4c8a-bc9d-a02bc4280985%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/23a8484a-dcfc-4c8a-bc9d-a02bc4280985%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/23a8484a-dcfc-4c8a-bc9d-a02bc4280985%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtjRiUhQPNL2i3TCr9ZNus%3DijMAZJ2J3P25vtHnxNGUag%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtjRiUhQPNL2i3TCr9ZNus%3DijMAZJ2J3P25vtHnxNGUag%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:01am UTC](https://discuss.elastic.co/t/how-to-scale-an-es-deployment-to-millions-of-tenants-with-different-data-schemas/19844/5 "2017-07-06T01:01:33Z")