# Architecture and performance question on searching small subsets of documents **URL:** \ **Category:** Elasticsearch\ **Created:** [July 15, 2014, 1:26pm UTC](https://discuss.elastic.co/t/architecture-and-performance-question-on-searching-small-subsets-of-documents/18677 "2014-07-15T13:26:35Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Mike\_Topper1](https://avatars.discourse-cdn.com/v4/letter/m/7bcc69/32.png) [@Mike\_Topper1](https://discuss.elastic.co/u/Mike_Topper1)\ **Post date:** [July 15, 2014, 1:26pm UTC](https://discuss.elastic.co/t/architecture-and-performance-question-on-searching-small-subsets-of-documents/18677/1 "2014-07-15T13:26:35Z")
Hello, I'm new elasticsearch, so this might be a stupid question but i'd love some input before I get started creating my elasticsearch cluster. Basically I will be indexing documents with a few fields (documents are pretty small in size). there are ~90million documents total. On the search side of things, each search will be limited by the small subset of documents that the user doing the search owns. my initial thought was to just have one large index for all documents and have a multi-value field that held the user ids of each user that owned that document. then when searching across the index i would do a filter query to limit by that user id. My only concern here is that this might be slow query times because you are always having to filter down by user id from a large data set to a very small subset (on average a user probably owns less than 1k documents). The other option I had is that i could create an index for each user and just index their documents into their index, but this would duplicate a massive amount of data and just seems hacky. Any suggestions? Thanks, Mike -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CALdNedL-sLM%3DyMWsHHzriBmMwfe08mxVG%3D%3D9tSwxLwiWzfAcyw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALdNedL-sLM%3DyMWsHHzriBmMwfe08mxVG%3D%3D9tSwxLwiWzfAcyw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [July 15, 2014, 1:32pm UTC](https://discuss.elastic.co/t/architecture-and-performance-question-on-searching-small-subsets-of-documents/18677/2 "2014-07-15T13:32:22Z")
Look at routing. It will help by limiting the searches to the shard with the user's data. Beyond that, you can generally trust the caching on filters to make this kind of use case quick. At least that is what I've seen on the mailing list. On Jul 15, 2014 9:27 AM, "Mike Topper" [topper@gmail.com](mailto:topper@gmail.com) wrote: > Hello, > > I'm new elasticsearch, so this might be a stupid question but i'd love > some input before I get started creating my elasticsearch cluster. > > Basically I will be indexing documents with a few fields (documents are > pretty small in size). there are ~90million documents total. > > On the search side of things, each search will be limited by the small > subset of documents that the user doing the search owns. > > my initial thought was to just have one large index for all documents and > have a multi-value field that held the user ids of each user that owned > that document. then when searching across the index i would do a filter > query to limit by that user id. My only concern here is that this might be > slow query times because you are always having to filter down by user id > from a large data set to a very small subset (on average a user probably > owns less than 1k documents). > > The other option I had is that i could create an index for each user and > just index their documents into their index, but this would duplicate a > massive amount of data and just seems hacky. > > Any suggestions? > > Thanks, > Mike > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CALdNedL-sLM%3DyMWsHHzriBmMwfe08mxVG%3D%3D9tSwxLwiWzfAcyw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALdNedL-sLM%3DyMWsHHzriBmMwfe08mxVG%3D%3D9tSwxLwiWzfAcyw%40mail.gmail.com) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3ccrAyDNZ8eR1xiG0fUvVw%3DAWY\_iXTGmik16cdDRKL3Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3ccrAyDNZ8eR1xiG0fUvVw%3DAWY_iXTGmik16cdDRKL3Q%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [July 16, 2014, 8:56am UTC](https://discuss.elastic.co/t/architecture-and-performance-question-on-searching-small-subsets-of-documents/18677/3 "2014-07-16T08:56:08Z")
Having a single large index is probably the best option, and will scale better when growing your base of users. I would recommend to watch the following video: [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/videos/big-data-search-and-analytics/). The part you are interested in starts at 13:20. On Tue, Jul 15, 2014 at 3:26 PM, Mike Topper [topper@gmail.com](mailto:topper@gmail.com) wrote: > Hello, > > I'm new elasticsearch, so this might be a stupid question but i'd love > some input before I get started creating my elasticsearch cluster. > > Basically I will be indexing documents with a few fields (documents are > pretty small in size). there are ~90million documents total. > > On the search side of things, each search will be limited by the small > subset of documents that the user doing the search owns. > > my initial thought was to just have one large index for all documents and > have a multi-value field that held the user ids of each user that owned > that document. then when searching across the index i would do a filter > query to limit by that user id. My only concern here is that this might be > slow query times because you are always having to filter down by user id > from a large data set to a very small subset (on average a user probably > owns less than 1k documents). > > The other option I had is that i could create an index for each user and > just index their documents into their index, but this would duplicate a > massive amount of data and just seems hacky. > > Any suggestions? > > Thanks, > Mike > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CALdNedL-sLM%3DyMWsHHzriBmMwfe08mxVG%3D%3D9tSwxLwiWzfAcyw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALdNedL-sLM%3DyMWsHHzriBmMwfe08mxVG%3D%3D9tSwxLwiWzfAcyw%40mail.gmail.com) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6cE4Nc66cP1TAyTjdWUqg0GiGj34ZpFcsjAZ8GR8cUjA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6cE4Nc66cP1TAyTjdWUqg0GiGj34ZpFcsjAZ8GR8cUjA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\ **Post date:** [July 16, 2014, 9:29am UTC](https://discuss.elastic.co/t/architecture-and-performance-question-on-searching-small-subsets-of-documents/18677/4 "2014-07-16T09:29:10Z")
Try the filter approach first and only if performance isn't good enough, look into other approaches. Lucene is quite fast at intersecting filters with large postings lists these days... Separate index per user is not only wasteful, because of the duplicated content, but will consume substantially more RAM/disk/file descriptors just because of the overhead required for an index. Mike McCandless [http://blog.mikemccandless.com](http://blog.mikemccandless.com) On Tue, Jul 15, 2014 at 9:26 AM, Mike Topper [topper@gmail.com](mailto:topper@gmail.com) wrote: > Hello, > > I'm new elasticsearch, so this might be a stupid question but i'd love > some input before I get started creating my elasticsearch cluster. > > Basically I will be indexing documents with a few fields (documents are > pretty small in size). there are ~90million documents total. > > On the search side of things, each search will be limited by the small > subset of documents that the user doing the search owns. > > my initial thought was to just have one large index for all documents and > have a multi-value field that held the user ids of each user that owned > that document. then when searching across the index i would do a filter > query to limit by that user id. My only concern here is that this might be > slow query times because you are always having to filter down by user id > from a large data set to a very small subset (on average a user probably > owns less than 1k documents). > > The other option I had is that i could create an index for each user and > just index their documents into their index, but this would duplicate a > massive amount of data and just seems hacky. > > Any suggestions? > > Thanks, > Mike > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CALdNedL-sLM%3DyMWsHHzriBmMwfe08mxVG%3D%3D9tSwxLwiWzfAcyw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALdNedL-sLM%3DyMWsHHzriBmMwfe08mxVG%3D%3D9tSwxLwiWzfAcyw%40mail.gmail.com) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAD7smRcj-ug82icWsPntUaXsYLJNwTcYgcHzD7tLk1SMi0PVQg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAD7smRcj-ug82icWsPntUaXsYLJNwTcYgcHzD7tLk1SMi0PVQg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:15am UTC](https://discuss.elastic.co/t/architecture-and-performance-question-on-searching-small-subsets-of-documents/18677/5 "2017-07-06T01:15:30Z")