# Index per company - any alternatives?

**URL:** <https://discuss.elastic.co/t/index-per-company-any-alternatives/11013>\
**Category:** Elasticsearch\
**Created:** [March 5, 2013, 12:39pm UTC](https://discuss.elastic.co/t/index-per-company-any-alternatives/11013 "2013-03-05T12:39:37Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![MoD](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mod/32/2459_2.png) [@MoD](https://discuss.elastic.co/u/MoD)\
**Post date:** [March 5, 2013, 12:39pm UTC](https://discuss.elastic.co/t/index-per-company-any-alternatives/11013/1 "2013-03-05T12:39:37Z")

</div>

Hi,

We are running a saas crm service. We set up elasticsearch to create an  
index per company (for example abc-company has its own index, xyz-company  
has its own index.).

But after 1000+ company we are suspecting this may not be a correct setup.

Especially when elasticsearch restarts (due to a failure) it starts with  
recovery and 1000+ index with 5 shards recovery takes forever (and %100  
cpu).

Any ideas for a correct setup?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [March 5, 2013, 12:49pm UTC](https://discuss.elastic.co/t/index-per-company-any-alternatives/11013/2 "2013-03-05T12:49:06Z")

</div>

You can over-allocate shards.

[https://groups.google.com/forum/#!msg/elasticsearch/49q-\_AgQCp8/MRol0t9asEcJ](https://groups.google.com/forum/#!msg/elasticsearch/49q-_AgQCp8/MRol0t9asEcJ)

Here are the docs for indices aliases with routing:

> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.

Jörg

Am 05.03.13 13:39, schrieb MoD:

> Hi,
> 
> We are running a saas crm service. We set up elasticsearch to create  
> an index per company (for example abc-company has its own index,  
> xyz-company has its own index.).
> 
> But after 1000+ company we are suspecting this may not be a correct  
> setup.
> 
> Especially when elasticsearch restarts (due to a failure) it starts  
> with recovery and 1000+ index with 5 shards recovery takes forever  
> (and %100 cpu).
> 
> Any ideas for a correct setup?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Michael\_Sick](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@Michael\_Sick](https://discuss.elastic.co/u/Michael_Sick)\
**Post date:** [March 5, 2013, 1:18pm UTC](https://discuss.elastic.co/t/index-per-company-any-alternatives/11013/3 "2013-03-05T13:18:00Z")

</div>

Can you explain more about the nature of the data? If it's not time based,  
as Joerg suggests using a single index with sharding and routing could be  
the answer.

On Tue, Mar 5, 2013 at 7:49 AM, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote:

> You can over-allocate shards.
> 
> [https://groups.google.com/\*\*forum/#!msg/elasticsearch/49q-](https://groups.google.com/**forum/#!msg/elasticsearch/49q-)\*\*  
> \_AgQCp8/MRol0t9asEcJ[https://groups.google.com/forum/#!msg/elasticsearch/49q-\_AgQCp8/MRol0t9asEcJ](https://groups.google.com/forum/#!msg/elasticsearch/49q-_AgQCp8/MRol0t9asEcJ)
> 
> Here are the docs for indices aliases with routing:  
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/**guide/reference/api/admin-)\*\*  
> indices-aliases.html[http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html](http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html)
> 
> Jörg
> 
> Am 05.03.13 13:39, schrieb MoD:
> 
> Hi,
> 
> > We are running a saas crm service. We set up elasticsearch to create an  
> > index per company (for example abc-company has its own index, xyz-company  
> > has its own index.).
> > 
> > But after 1000+ company we are suspecting this may not be a correct setup.
> > 
> > Especially when elasticsearch restarts (due to a failure) it starts with  
> > recovery and 1000+ index with 5 shards recovery takes forever (and %100  
> > cpu).
> > 
> > Any ideas for a correct setup?
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com)  
> .  
> For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)  
> .

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![MoD](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mod/32/2459_2.png) [@MoD](https://discuss.elastic.co/u/MoD)\
**Post date:** [March 5, 2013, 6:20pm UTC](https://discuss.elastic.co/t/index-per-company-any-alternatives/11013/4 "2013-03-05T18:20:02Z")

</div>

The data is the contacts/companies/notes of each company (domain, user). We  
are using elasticsearch to index the data and full text search the data on  
the site.

It is not timebased and will remain searchable as long as the company  
wishes to use the product.

At first we thought that in order to limit the search within the company we  
should use index per company.

But after the company/user base grew, we found out that recovery of indexes  
takes too long. By the way, that is the sole reason (the recovery phase) we  
wish to change the setup.

On Tuesday, 5 March 2013 15:18:00 UTC+2, Michael Sick wrote:

> Can you explain more about the nature of the data? If it's not time based,  
> as Joerg suggests using a single index with sharding and routing could be  
> the answer.
> 
> On Tue, Mar 5, 2013 at 7:49 AM, Jörg Prante \<[joerg...@gmail.com](mailto:joerg...@gmail.com)\<javascript:\>
> 
> > wrote:
> 
> > You can over-allocate shards.
> > 
> > [https://groups.google.com/\*\*forum/#!msg/elasticsearch/49q-](https://groups.google.com/**forum/#!msg/elasticsearch/49q-)\*\*  
> > \_AgQCp8/MRol0t9asEcJ[https://groups.google.com/forum/#!msg/elasticsearch/49q-\_AgQCp8/MRol0t9asEcJ](https://groups.google.com/forum/#!msg/elasticsearch/49q-_AgQCp8/MRol0t9asEcJ)
> > 
> > Here are the docs for indices aliases with routing:  
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/**guide/reference/api/admin-)\*\*  
> > indices-aliases.html[http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html](http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html)
> > 
> > Jörg
> > 
> > Am 05.03.13 13:39, schrieb MoD:
> > 
> > Hi,
> > 
> > > We are running a saas crm service. We set up elasticsearch to create an  
> > > index per company (for example abc-company has its own index, xyz-company  
> > > has its own index.).
> > > 
> > > But after 1000+ company we are suspecting this may not be a correct  
> > > setup.
> > > 
> > > Especially when elasticsearch restarts (due to a failure) it starts with  
> > > recovery and 1000+ index with 5 shards recovery takes forever (and %100  
> > > cpu).
> > > 
> > > Any ideas for a correct setup?
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com) \<javascript:\>.  
> > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)  
> > .

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\
**Post date:** [March 5, 2013, 7:25pm UTC](https://discuss.elastic.co/t/index-per-company-any-alternatives/11013/5 "2013-03-05T19:25:50Z")

</div>

Are you able to figure out what is eating up the most time during recovery?  
If you set index.gateway to DEBUG log level you should be able to get those  
details.

One alternative solution is to tweak the index.translog.flush\_threshold in  
the config file. I deal with a decent number of indexes (less than you do,  
though) and moving this from the default of 5000 down to 1000 helped our  
recovery times. It's a tradeoff, since you will have more merges, but if  
your indexing volume is small won't make a difference.

This should only require a cluster restart instead of a full data rebuild.

That being said, one index with routing key on company will definitely  
help.

Best Regards,  
Paul

On Tuesday, March 5, 2013 11:20:02 AM UTC-7, MoD wrote:

> The data is the contacts/companies/notes of each company (domain, user).  
> We are using elasticsearch to index the data and full text search the data  
> on the site.
> 
> It is not timebased and will remain searchable as long as the company  
> wishes to use the product.
> 
> At first we thought that in order to limit the search within the company  
> we should use index per company.
> 
> But after the company/user base grew, we found out that recovery of  
> indexes takes too long. By the way, that is the sole reason (the recovery  
> phase) we wish to change the setup.
> 
> On Tuesday, 5 March 2013 15:18:00 UTC+2, Michael Sick wrote:
> 
> > Can you explain more about the nature of the data? If it's not time  
> > based, as Joerg suggests using a single index with sharding and routing  
> > could be the answer.
> > 
> > On Tue, Mar 5, 2013 at 7:49 AM, Jörg Prante [joerg...@gmail.com](mailto:joerg...@gmail.com) wrote:
> > 
> > > You can over-allocate shards.
> > > 
> > > [https://groups.google.com/\*\*forum/#!msg/elasticsearch/49q-](https://groups.google.com/**forum/#!msg/elasticsearch/49q-)\*\*  
> > > \_AgQCp8/MRol0t9asEcJ[https://groups.google.com/forum/#!msg/elasticsearch/49q-\_AgQCp8/MRol0t9asEcJ](https://groups.google.com/forum/#!msg/elasticsearch/49q-_AgQCp8/MRol0t9asEcJ)
> > > 
> > > Here are the docs for indices aliases with routing:  
> > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/**guide/reference/api/admin-)\*\*  
> > > indices-aliases.html[http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html](http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html)
> > > 
> > > Jörg
> > > 
> > > Am 05.03.13 13:39, schrieb MoD:
> > > 
> > > Hi,
> > > 
> > > > We are running a saas crm service. We set up elasticsearch to create an  
> > > > index per company (for example abc-company has its own index, xyz-company  
> > > > has its own index.).
> > > > 
> > > > But after 1000+ company we are suspecting this may not be a correct  
> > > > setup.
> > > > 
> > > > Especially when elasticsearch restarts (due to a failure) it starts  
> > > > with recovery and 1000+ index with 5 shards recovery takes forever (and  
> > > > %100 cpu).
> > > > 
> > > > Any ideas for a correct setup?
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google  
> > > Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send  
> > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com).  
> > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)  
> > > .

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![MoD](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mod/32/2459_2.png) [@MoD](https://discuss.elastic.co/u/MoD)\
**Post date:** [February 22, 2016, 9:43am UTC](https://discuss.elastic.co/t/index-per-company-any-alternatives/11013/6 "2016-02-22T09:43:04Z")

</div>

We changed the setup to a single index (ie. master) and multiple aliases as in  
[https://www.elastic.co/guide/en/elasticsearch/guide/current/faking-it.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/faking-it.html)

Index creation was taking to long and restarting too. This way all our problems are solved.

Thanks for the help.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:14pm UTC](https://discuss.elastic.co/t/index-per-company-any-alternatives/11013/7 "2017-07-05T23:14:30Z")

</div>