# Best solution for keeping data of many clients

**URL:** <https://discuss.elastic.co/t/best-solution-for-keeping-data-of-many-clients/8673>\
**Category:** Elasticsearch\
**Created:** [August 8, 2012, 2:03pm UTC](https://discuss.elastic.co/t/best-solution-for-keeping-data-of-many-clients/8673 "2012-08-08T14:03:44Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Marcin\_Dojwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_dojwa/32/2307_2.png) [@Marcin\_Dojwa](https://discuss.elastic.co/u/Marcin_Dojwa)\
**Post date:** [August 8, 2012, 2:03pm UTC](https://discuss.elastic.co/t/best-solution-for-keeping-data-of-many-clients/8673/1 "2012-08-08T14:03:44Z")

</div>

Hi,

I have the following problem. I have to design system based on ES keeping  
the of many clients with many documents. Here are the details.

1. It must keep data of up to 1mln clients
2. I have 3 types of data (eg. chats, login\_data, left\_messages)
3. Each client has up to 10mln documents of each type.

My current design looks like this:  
/\<client\_number\>/\<document\_type\>/\<document\_id\> (eg. /1231/chats/XSAuxaS)  
It gives me up to 1mln indices created. Currently with 640 indices I have  
problem with "too many open files" while I have set it up to 65535. I  
believe that this is not the right way to design this right ? 🙂

So I have 2 solutions:

1. Design it like this: /\<document\_type\>/\<client\_number\>/\<document\_id\> -  
this gives me 3 indices but up to 1mln document types. Would this still  
cause "too many open files" problem or any other problems ?
2. Put all clients' data to one document type. Eg. put all clients' chats  
to /anything/chats/\<document\_id\> but this gives me really huge number of  
documents in a single index and document type.

Important thing is that I always query for a specified \<client\_number\> and  
\<document\_type\>, so I do not need it in a common bucket.

So, what is the best solution using ES in this case ? 🙂 If you need more  
information let me know. Thanks for any help.

Best regards.  
Marcin Dojwa.

---

<div class="post-metadata">

**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\
**Post date:** [August 8, 2012, 2:16pm UTC](https://discuss.elastic.co/t/best-solution-for-keeping-data-of-many-clients/8673/2 "2012-08-08T14:16:01Z")

</div>

Hi,

have you seen Shay's talk [1] where he explains index aliasing [2] with  
routing and filter feature?

[1]  
[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/videos/2012/06/05/big-data-search-and-analytics.html)(sounds  
like "users" data flow [starting at slide 20/44] is what you are  
looking for?)  
[2]  
[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html)

Regards,  
Lukas

On Wed, Aug 8, 2012 at 4:03 PM, Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)wrote:

> Hi,
> 
> I have the following problem. I have to design system based on ES keeping  
> the of many clients with many documents. Here are the details.
> 
> 1. It must keep data of up to 1mln clients
> 2. I have 3 types of data (eg. chats, login\_data, left\_messages)
> 3. Each client has up to 10mln documents of each type.
> 
> My current design looks like this:  
> /\<client\_number\>/\<document\_type\>/\<document\_id\> (eg. /1231/chats/XSAuxaS)  
> It gives me up to 1mln indices created. Currently with 640 indices I have  
> problem with "too many open files" while I have set it up to 65535. I  
> believe that this is not the right way to design this right ? 🙂
> 
> So I have 2 solutions:
> 
> 1. Design it like this: /\<document\_type\>/\<client\_number\>/\<document\_id\> -  
> this gives me 3 indices but up to 1mln document types. Would this still  
> cause "too many open files" problem or any other problems ?
> 2. Put all clients' data to one document type. Eg. put all clients' chats  
> to /anything/chats/\<document\_id\> but this gives me really huge number of  
> documents in a single index and document type.
> 
> Important thing is that I always query for a specified \<client\_number\> and  
> \<document\_type\>, so I do not need it in a common bucket.
> 
> So, what is the best solution using ES in this case ? 🙂 If you need more  
> information let me know. Thanks for any help.
> 
> Best regards.  
> Marcin Dojwa.

---

<div class="post-metadata">

**Author:** ![Marcin\_Dojwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_dojwa/32/2307_2.png) [@Marcin\_Dojwa](https://discuss.elastic.co/u/Marcin_Dojwa)\
**Post date:** [August 8, 2012, 2:44pm UTC](https://discuss.elastic.co/t/best-solution-for-keeping-data-of-many-clients/8673/3 "2012-08-08T14:44:49Z")

</div>

Thank you, I will check this out.

Best regards.

2012/8/8 Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)

> Hi,
> 
> have you seen Shay's talk [1] where he explains index aliasing [2] with  
> routing and filter feature?
> 
> [1]  
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/videos/2012/06/05/big-data-search-and-analytics.html)(sounds like "users" data flow [starting at slide 20/44] is what you are  
> looking for?)  
> [2]  
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html)
> 
> Regards,  
> Lukas
> 
> On Wed, Aug 8, 2012 at 4:03 PM, Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)wrote:
> 
> > Hi,
> > 
> > I have the following problem. I have to design system based on ES keeping  
> > the of many clients with many documents. Here are the details.
> > 
> > 1. It must keep data of up to 1mln clients
> > 2. I have 3 types of data (eg. chats, login\_data, left\_messages)
> > 3. Each client has up to 10mln documents of each type.
> > 
> > My current design looks like this:  
> > /\<client\_number\>/\<document\_type\>/\<document\_id\> (eg. /1231/chats/XSAuxaS)  
> > It gives me up to 1mln indices created. Currently with 640 indices I have  
> > problem with "too many open files" while I have set it up to 65535. I  
> > believe that this is not the right way to design this right ? 🙂
> > 
> > So I have 2 solutions:
> > 
> > 1. Design it like this: /\<document\_type\>/\<client\_number\>/\<document\_id\> -  
> > this gives me 3 indices but up to 1mln document types. Would this still  
> > cause "too many open files" problem or any other problems ?
> > 2. Put all clients' data to one document type. Eg. put all clients' chats  
> > to /anything/chats/\<document\_id\> but this gives me really huge number of  
> > documents in a single index and document type.
> > 
> > Important thing is that I always query for a specified \<client\_number\>  
> > and \<document\_type\>, so I do not need it in a common bucket.
> > 
> > So, what is the best solution using ES in this case ? 🙂 If you need more  
> > information let me know. Thanks for any help.
> > 
> > Best regards.  
> > Marcin Dojwa.

---

<div class="post-metadata">

**Author:** ![Marcin\_Dojwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_dojwa/32/2307_2.png) [@Marcin\_Dojwa](https://discuss.elastic.co/u/Marcin_Dojwa)\
**Post date:** [August 8, 2012, 3:27pm UTC](https://discuss.elastic.co/t/best-solution-for-keeping-data-of-many-clients/8673/4 "2012-08-08T15:27:32Z")

</div>

Thank you very much, I think this solves all my problems 🙂

Best regards.

2012/8/8 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)

> Thank you, I will check this out.
> 
> Best regards.
> 
> 2012/8/8 Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)
> 
> > Hi,
> > 
> > have you seen Shay's talk [1] where he explains index aliasing [2] with  
> > routing and filter feature?
> > 
> > [1]  
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/videos/2012/06/05/big-data-search-and-analytics.html)(sounds like "users" data flow [starting at slide 20/44] is what you are  
> > looking for?)  
> > [2]  
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html)
> > 
> > Regards,  
> > Lukas
> > 
> > On Wed, Aug 8, 2012 at 4:03 PM, Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)wrote:
> > 
> > > Hi,
> > > 
> > > I have the following problem. I have to design system based on ES  
> > > keeping the of many clients with many documents. Here are the details.
> > > 
> > > 1. It must keep data of up to 1mln clients
> > > 2. I have 3 types of data (eg. chats, login\_data, left\_messages)
> > > 3. Each client has up to 10mln documents of each type.
> > > 
> > > My current design looks like this:  
> > > /\<client\_number\>/\<document\_type\>/\<document\_id\> (eg. /1231/chats/XSAuxaS)  
> > > It gives me up to 1mln indices created. Currently with 640 indices I  
> > > have problem with "too many open files" while I have set it up to 65535. I  
> > > believe that this is not the right way to design this right ? 🙂
> > > 
> > > So I have 2 solutions:
> > > 
> > > 1. Design it like this: /\<document\_type\>/\<client\_number\>/\<document\_id\> -  
> > > this gives me 3 indices but up to 1mln document types. Would this still  
> > > cause "too many open files" problem or any other problems ?
> > > 2. Put all clients' data to one document type. Eg. put all clients'  
> > > chats to /anything/chats/\<document\_id\> but this gives me really huge number  
> > > of documents in a single index and document type.
> > > 
> > > Important thing is that I always query for a specified \<client\_number\>  
> > > and \<document\_type\>, so I do not need it in a common bucket.
> > > 
> > > So, what is the best solution using ES in this case ? 🙂 If you need  
> > > more information let me know. Thanks for any help.
> > > 
> > > Best regards.  
> > > Marcin Dojwa.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:17am UTC](https://discuss.elastic.co/t/best-solution-for-keeping-data-of-many-clients/8673/5 "2017-07-06T03:17:07Z")

</div>

Closed on July 6, 2017, 3:17am UTC