# Need suggestion on sharding for efficiency **URL:** \ **Category:** Elasticsearch\ **Created:** [March 25, 2016, 12:07am UTC](https://discuss.elastic.co/t/need-suggestion-on-sharding-for-efficiency/45407 "2016-03-25T00:07:43Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![rahulcse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahulcse/32/12158_2.png) [@rahulcse](https://discuss.elastic.co/u/rahulcse)\ **Post date:** [March 25, 2016, 12:07am UTC](https://discuss.elastic.co/t/need-suggestion-on-sharding-for-efficiency/45407/1 "2016-03-25T00:07:43Z")
We are trying to design search engine for our customers. We have many customers, some have small data, some have medium size data and some have really big data. We are trying to go for one index per customer. Also for small customers we want less shards and for medium one and larger ones we want an optimal number of shards. Can anyone suggest us what is the right approach for this? Please share your views it will be helpful. Thanks in advance for your help. Regards, Rahul ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 25, 2016, 12:26am UTC](https://discuss.elastic.co/t/need-suggestion-on-sharding-for-efficiency/45407/2 "2016-03-25T00:26:12Z")
You are better off having a small users index, that leverages routing. Then the same for medium users, but more shards. Then the large users get their own index. ---
**Author:** ![rahulcse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahulcse/32/12158_2.png) [@rahulcse](https://discuss.elastic.co/u/rahulcse)\ **Post date:** [March 25, 2016, 12:43am UTC](https://discuss.elastic.co/t/need-suggestion-on-sharding-for-efficiency/45407/3 "2016-03-25T00:43:54Z")
Hi Mark, thank you for your time and answering. How can we mix data from different small clients into one index? Will it create a security issue, as it is very sensitive data? If we go for small number of shards per index per small customers what problems we can face? The system is high on query volume and low on writes and updates. Regards, Rahul ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 25, 2016, 12:44am UTC](https://discuss.elastic.co/t/need-suggestion-on-sharding-for-efficiency/45407/4 "2016-03-25T00:44:56Z")
> [@rahulcse](#): > > How can we mix data from different small clients into one index? Will it create a security issue, as it is very sensitive data? Shield can help you deal with this. > [@rahulcse](#): > > If we go for small number of shards per index per small customers what problems we can face? Lots of small indices/shards wastes system resources. ---
**Author:** ![rahulcse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahulcse/32/12158_2.png) [@rahulcse](https://discuss.elastic.co/u/rahulcse)\ **Post date:** [March 25, 2016, 12:47am UTC](https://discuss.elastic.co/t/need-suggestion-on-sharding-for-efficiency/45407/5 "2016-03-25T00:47:32Z")
Hi Mark, we have many machines at our disposal, so thats not a big issue. Also with shield usage we had to disable the java security manager in ES as we are making external calls to other services for social ranking, will that be an issue with Shield? Regards, Rahul ---
**Author:** ![rahulcse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahulcse/32/12158_2.png) [@rahulcse](https://discuss.elastic.co/u/rahulcse)\ **Post date:** [March 25, 2016, 1:11am UTC](https://discuss.elastic.co/t/need-suggestion-on-sharding-for-efficiency/45407/6 "2016-03-25T01:11:30Z")
Also if we have many clients in one index, will scoring and IDF things will get corrupted and will give us some different ranking. Please share your thoughts on this also. Thanks for all your help. Regards, Rahul ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 25, 2016, 1:15am UTC](https://discuss.elastic.co/t/need-suggestion-on-sharding-for-efficiency/45407/7 "2016-03-25T01:15:37Z")
How many users do you expect to support? Are you in control of mappings? ---
**Author:** ![rahulcse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahulcse/32/12158_2.png) [@rahulcse](https://discuss.elastic.co/u/rahulcse)\ **Post date:** [March 25, 2016, 1:20am UTC](https://discuss.elastic.co/t/need-suggestion-on-sharding-for-efficiency/45407/8 "2016-03-25T01:20:01Z")
Each client is a unique id we have. And there are around 300 small clients (each client has around 1000's work force who will query our search clusters). There are around 400 medium clients and around 40 big ones. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 25, 2016, 1:33am UTC](https://discuss.elastic.co/t/need-suggestion-on-sharding-for-efficiency/45407/9 "2016-03-25T01:33:19Z")
Having an index per user does tend to scale badly and lots of small indikes waste system resources due to the overhead associated with each shard. If you however expect less than a thousand users, going with a single, separate index per user may actually work. Small users should probably have indices with just a single shard, and this may, depending on data volumes, also apply to medium users too. Best way to find out is to test under as realistic conditions as possible. Make sure that you use Elasticsearch 2.x so you can benefit from delta cluster state updates. ---
**Author:** ![rahulcse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahulcse/32/12158_2.png) [@rahulcse](https://discuss.elastic.co/u/rahulcse)\ **Post date:** [March 25, 2016, 1:40am UTC](https://discuss.elastic.co/t/need-suggestion-on-sharding-for-efficiency/45407/10 "2016-03-25T01:40:18Z")
Hi Christian, thank you for your advice. Can you please tell me how the IDF scoring is effected if we have multiple customers in one index? Regards, Rahul ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 25, 2016, 1:51am UTC](https://discuss.elastic.co/t/need-suggestion-on-sharding-for-efficiency/45407/11 "2016-03-25T01:51:06Z")
Scoring is per shard, so if you use routing then each customers scoring will be relevant to their own. ---
**Author:** ![rahulcse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahulcse/32/12158_2.png) [@rahulcse](https://discuss.elastic.co/u/rahulcse)\ **Post date:** [March 25, 2016, 4:33am UTC](https://discuss.elastic.co/t/need-suggestion-on-sharding-for-efficiency/45407/12 "2016-03-25T04:33:32Z")
Thanks a lot for your input, they really open horizon for us. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 25, 2016, 8:52am UTC](https://discuss.elastic.co/t/need-suggestion-on-sharding-for-efficiency/45407/13 "2016-03-25T08:52:08Z")
When you use routing, all documents belonging to a customer will be located in the same shard, but there will be multiple users per shard, so it could impact storing. ---
**Author:** ![rahulcse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahulcse/32/12158_2.png) [@rahulcse](https://discuss.elastic.co/u/rahulcse)\ **Post date:** [March 25, 2016, 10:15pm UTC](https://discuss.elastic.co/t/need-suggestion-on-sharding-for-efficiency/45407/14 "2016-03-25T22:15:49Z")
Routing if fine when u want to store specific data on some shards, but here data is very general apart from language there is no way to differentiate in routing. The queries to the system are general searching the document by title and body. Do you have more inputs on routing? Thanks a lot for all your help. Regards, Rahul ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:05pm UTC](https://discuss.elastic.co/t/need-suggestion-on-sharding-for-efficiency/45407/15 "2017-07-05T23:05:03Z")