# Search Across Multiple Indexes **URL:** \ **Category:** Elasticsearch\ **Created:** [May 11, 2012, 1:39am UTC](https://discuss.elastic.co/t/search-across-multiple-indexes/7647 "2012-05-11T01:39:05Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Kinesh\_Satiya](https://avatars.discourse-cdn.com/v4/letter/k/a88e4f/32.png) [@Kinesh\_Satiya](https://discuss.elastic.co/u/Kinesh_Satiya)\ **Post date:** [May 11, 2012, 1:39am UTC](https://discuss.elastic.co/t/search-across-multiple-indexes/7647/1 "2012-05-11T01:39:05Z")
Hi, I have two indexes, they basically index notes. A note has a noteid, title description, userid. So my first index is notes(index)/note(mapping) with noteid Second is note content which has list of note ids and note content and id for this is sha1(title) and list of userids who have this note in common. So note title is common between these two indexes. All I want to do is search across both of theses indexes, with individual boost factors for title description and content with a given keyword. I know how to do the individual searches with bool and adding boost factors. Currently I first search on first index with the given keyword and then search on next index for content and then merge both the search results. But I do want to search across both of these indexes together so that I can have relevant order. Thanks ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 13, 2012, 12:06pm UTC](https://discuss.elastic.co/t/search-across-multiple-indexes/7647/2 "2012-05-13T12:06:12Z")
I don't really understand what is the problem you have with search across the two indices... On Fri, May 11, 2012 at 4:39 AM, Kinesh Satiya [ksatiya@mysocialcloud.com](mailto:ksatiya@mysocialcloud.com)wrote: > Hi, > > I have two indexes, they basically index notes. A note has a noteid, > title description, userid. > > So my first index is notes(index)/note(mapping) with noteid > > Second is note content which has list of note ids and note content and > id for this is sha1(title) and list of userids who have this note in > common. So note title is common between these two indexes. > > All I want to do is search across both of theses indexes, with > individual boost factors for title description and content with a > given keyword. > > I know how to do the individual searches with bool and adding boost > factors. > Currently I first search on first index with the given keyword and > then search on next index for content and then merge both the search > results. > > But I do want to search across both of these indexes together so that > I can have relevant order. > > Thanks ---
**Author:** ![Crwe](https://avatars.discourse-cdn.com/v4/letter/c/898d66/32.png) [@Crwe](https://discuss.elastic.co/u/Crwe)\ **Post date:** [May 13, 2012, 5:40pm UTC](https://discuss.elastic.co/t/search-across-multiple-indexes/7647/3 "2012-05-13T17:40:48Z")
On May 13, 2:06 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > I don't really understand what is the problem you have with search across > the two indices... For one, the relevance scores from different indexes seem to be uncomparable. In particular, the idf values seem to be calculated for each index separately. This means that when results from two indexes are merged into a single ranked list, it's really comparing apples to oranges. > On Fri, May 11, 2012 at 4:39 AM, Kinesh Satiya [ksat...@mysocialcloud.com](mailto:ksat...@mysocialcloud.com)wrote: > > > Hi, > > > I have two indexes, they basically index notes. A note has a noteid, > > title description, userid. > > > So my first index is notes(index)/note(mapping) with noteid > > > Second is note content which has list of note ids and note content and > > id for this is sha1(title) and list of userids who have this note in > > common. So note title is common between these two indexes. > > > All I want to do is search across both of theses indexes, with > > individual boost factors for title description and content with a > > given keyword. > > > I know how to do the individual searches with bool and adding boost > > factors. > > Currently I first search on first index with the given keyword and > > then search on next index for content and then merge both the search > > results. > > > But I do want to search across both of these indexes together so that > > I can have relevant order. > > > Thanks ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 15, 2012, 8:26pm UTC](https://discuss.elastic.co/t/search-across-multiple-indexes/7647/4 "2012-05-15T20:26:57Z")
Actually, the idf is computed per shard (by default). You can use a search type set to dfs\_query\_then\_fetch, which will add another phase to teh search to compute distributed frequencies (but note it wil be slower). On Sun, May 13, 2012 at 8:40 PM, Crwe [tester.testerus@gmail.com](mailto:tester.testerus@gmail.com) wrote: > On May 13, 2:06 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > I don't really understand what is the problem you have with search across > > the two indices... > > For one, the relevance scores from different indexes seem to be > uncomparable. > > In particular, the idf values seem to be calculated for each index > separately. This means that when results from two indexes are merged > into a single ranked list, it's really comparing apples to oranges. > > > On Fri, May 11, 2012 at 4:39 AM, Kinesh Satiya \< > > [ksat...@mysocialcloud.com](mailto:ksat...@mysocialcloud.com)\>wrote: > > > > > Hi, > > > > > I have two indexes, they basically index notes. A note has a noteid, > > > title description, userid. > > > > > So my first index is notes(index)/note(mapping) with noteid > > > > > Second is note content which has list of note ids and note content and > > > id for this is sha1(title) and list of userids who have this note in > > > common. So note title is common between these two indexes. > > > > > All I want to do is search across both of theses indexes, with > > > individual boost factors for title description and content with a > > > given keyword. > > > > > I know how to do the individual searches with bool and adding boost > > > factors. > > > Currently I first search on first index with the given keyword and > > > then search on next index for content and then merge both the search > > > results. > > > > > But I do want to search across both of these indexes together so that > > > I can have relevant order. > > > > > Thanks ---
**Author:** ![Crwe](https://avatars.discourse-cdn.com/v4/letter/c/898d66/32.png) [@Crwe](https://discuss.elastic.co/u/Crwe)\ **Post date:** [May 20, 2012, 8:24am UTC](https://discuss.elastic.co/t/search-across-multiple-indexes/7647/5 "2012-05-20T08:24:32Z")
Oh, so this can happen for a single index, too? I would be interested in using the `dfs_query_then_fetch` then -- what does "slower" mean in this context? Asymptotically slower? Constant factor (x #shards) slower? More generally, is there a resource page for ES which describes performance implications of various settings (incl. `refresh`)? Thank you in advance. On May 15, 10:26 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > Actually, the idf is computed per shard (by default). You can use a search > type set to dfs\_query\_then\_fetch, which will add another phase to teh > search to compute distributed frequencies (but note it wil be slower). > > On Sun, May 13, 2012 at 8:40 PM, Crwe [tester.teste...@gmail.com](mailto:tester.teste...@gmail.com) wrote: > > > On May 13, 2:06 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > > I don't really understand what is the problem you have with search across > > > the two indices... > > > For one, the relevance scores from different indexes seem to be > > uncomparable. > > > In particular, the idf values seem to be calculated for each index > > separately. This means that when results from two indexes are merged > > into a single ranked list, it's really comparing apples to oranges. > > > > On Fri, May 11, 2012 at 4:39 AM, Kinesh Satiya \< > > > [ksat...@mysocialcloud.com](mailto:ksat...@mysocialcloud.com)\>wrote: > > > > > Hi, > > > > > I have two indexes, they basically index notes. A note has a noteid, > > > > title description, userid. > > > > > So my first index is notes(index)/note(mapping) with noteid > > > > > Second is note content which has list of note ids and note content and > > > > id for this is sha1(title) and list of userids who have this note in > > > > common. So note title is common between these two indexes. > > > > > All I want to do is search across both of theses indexes, with > > > > individual boost factors for title description and content with a > > > > given keyword. > > > > > I know how to do the individual searches with bool and adding boost > > > > factors. > > > > Currently I first search on first index with the given keyword and > > > > then search on next index for content and then merge both the search > > > > results. > > > > > But I do want to search across both of these indexes together so that > > > > I can have relevant order. > > > > > Thanks ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 20, 2012, 10:00pm UTC](https://discuss.elastic.co/t/search-across-multiple-indexes/7647/6 "2012-05-20T22:00:55Z")
Hard to say how much slower, effectively, what happens is that the DFS phase goes to all the relevant shards, and extracts the terms associated with the query and their frequencies, it then aggregates those and execute the search on each shard with the aggregated value. On Sun, May 20, 2012 at 10:24 AM, Crwe [tester.testerus@gmail.com](mailto:tester.testerus@gmail.com) wrote: > Oh, so this can happen for a single index, too? I would be interested > in using the `dfs_query_then_fetch` then -- what does "slower" mean in > this context? Asymptotically slower? Constant factor (x #shards) > slower? > > More generally, is there a resource page for ES which describes > performance implications of various settings (incl. `refresh`)? > > Thank you in advance. > > On May 15, 10:26 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > Actually, the idf is computed per shard (by default). You can use a > > search > > type set to dfs\_query\_then\_fetch, which will add another phase to teh > > search to compute distributed frequencies (but note it wil be slower). > > > > On Sun, May 13, 2012 at 8:40 PM, Crwe [tester.teste...@gmail.com](mailto:tester.teste...@gmail.com) wrote: > > > > > On May 13, 2:06 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > > > > I don't really understand what is the problem you have with search > > > > across > > > > the two indices... > > > > > For one, the relevance scores from different indexes seem to be > > > uncomparable. > > > > > In particular, the idf values seem to be calculated for each index > > > separately. This means that when results from two indexes are merged > > > into a single ranked list, it's really comparing apples to oranges. > > > > > > On Fri, May 11, 2012 at 4:39 AM, Kinesh Satiya \< > > > > [ksat...@mysocialcloud.com](mailto:ksat...@mysocialcloud.com)\>wrote: > > > > > > > Hi, > > > > > > > I have two indexes, they basically index notes. A note has a > > > > > noteid, > > > > > title description, userid. > > > > > > > So my first index is notes(index)/note(mapping) with noteid > > > > > > > Second is note content which has list of note ids and note content > > > > > and > > > > > id for this is sha1(title) and list of userids who have this note > > > > > in > > > > > common. So note title is common between these two indexes. > > > > > > > All I want to do is search across both of theses indexes, with > > > > > individual boost factors for title description and content with a > > > > > given keyword. > > > > > > > I know how to do the individual searches with bool and adding boost > > > > > factors. > > > > > Currently I first search on first index with the given keyword and > > > > > then search on next index for content and then merge both the > > > > > search > > > > > results. > > > > > > > But I do want to search across both of these indexes together so > > > > > that > > > > > I can have relevant order. > > > > > > > Thanks ---
**Author:** ![Crwe](https://avatars.discourse-cdn.com/v4/letter/c/898d66/32.png) [@Crwe](https://discuss.elastic.co/u/Crwe)\ **Post date:** [May 21, 2012, 3:55pm UTC](https://discuss.elastic.co/t/search-across-multiple-indexes/7647/7 "2012-05-21T15:55:54Z")
Ok, that sounds like each shard will be accessed twice, instead of once. Unless the aggregation phase itself is the bottleneck, this means about 0.5x performance hit (independent of #shards), which is quite acceptable for me. Good news. Thanks again Shay. On May 21, 12:00 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > Hard to say how much slower, effectively, what happens is that the DFS > phase goes to all the relevant shards, and extracts the terms associated > with the query and their frequencies, it then aggregates those and execute > the search on each shard with the aggregated value. > > On Sun, May 20, 2012 at 10:24 AM, Crwe [tester.teste...@gmail.com](mailto:tester.teste...@gmail.com) wrote: > > > Oh, so this can happen for a single index, too? I would be interested > > in using the `dfs_query_then_fetch` then -- what does "slower" mean in > > this context? Asymptotically slower? Constant factor (x #shards) > > slower? > > > More generally, is there a resource page for ES which describes > > performance implications of various settings (incl. `refresh`)? > > > Thank you in advance. > > > On May 15, 10:26 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > > Actually, the idf is computed per shard (by default). You can use a > > > search > > > type set to dfs\_query\_then\_fetch, which will add another phase to teh > > > search to compute distributed frequencies (but note it wil be slower). > > > > On Sun, May 13, 2012 at 8:40 PM, Crwe [tester.teste...@gmail.com](mailto:tester.teste...@gmail.com) wrote: > > > > > > > On May 13, 2:06 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > > > > > > I don't really understand what is the problem you have with search > > > > > across > > > > > the two indices... > > > > > For one, the relevance scores from different indexes seem to be > > > > uncomparable. > > > > > In particular, the idf values seem to be calculated for each index > > > > separately. This means that when results from two indexes are merged > > > > into a single ranked list, it's really comparing apples to oranges. > > > > > > On Fri, May 11, 2012 at 4:39 AM, Kinesh Satiya \< > > > > > [ksat...@mysocialcloud.com](mailto:ksat...@mysocialcloud.com)\>wrote: > > > > > > > Hi, > > > > > > > I have two indexes, they basically index notes. A note has a > > > > > > noteid, > > > > > > title description, userid. > > > > > > > So my first index is notes(index)/note(mapping) with noteid > > > > > > > Second is note content which has list of note ids and note content > > > > > > and > > > > > > id for this is sha1(title) and list of userids who have this note > > > > > > in > > > > > > common. So note title is common between these two indexes. > > > > > > > All I want to do is search across both of theses indexes, with > > > > > > individual boost factors for title description and content with a > > > > > > given keyword. > > > > > > > I know how to do the individual searches with bool and adding boost > > > > > > factors. > > > > > > Currently I first search on first index with the given keyword and > > > > > > then search on next index for content and then merge both the > > > > > > search > > > > > > results. > > > > > > > But I do want to search across both of these indexes together so > > > > > > that > > > > > > I can have relevant order. > > > > > > > Thanks ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 23, 2012, 9:37pm UTC](https://discuss.elastic.co/t/search-across-multiple-indexes/7647/8 "2012-05-23T21:37:27Z")
Since there is also the cost of "search" itself, it won't be 0.5x, it will probably be lower, faster network here is important. On Mon, May 21, 2012 at 5:55 PM, Crwe [tester.testerus@gmail.com](mailto:tester.testerus@gmail.com) wrote: > Ok, that sounds like each shard will be accessed twice, instead of > once. Unless the aggregation phase itself is the bottleneck, this > means about 0.5x performance hit (independent of #shards), which is > quite acceptable for me. Good news. > > Thanks again Shay. > > On May 21, 12:00 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > Hard to say how much slower, effectively, what happens is that the DFS > > phase goes to all the relevant shards, and extracts the terms associated > > with the query and their frequencies, it then aggregates those and > > execute > > the search on each shard with the aggregated value. > > > > On Sun, May 20, 2012 at 10:24 AM, Crwe [tester.teste...@gmail.com](mailto:tester.teste...@gmail.com) > > wrote: > > > > > Oh, so this can happen for a single index, too? I would be interested > > > in using the `dfs_query_then_fetch` then -- what does "slower" mean in > > > this context? Asymptotically slower? Constant factor (x #shards) > > > slower? > > > > > More generally, is there a resource page for ES which describes > > > performance implications of various settings (incl. `refresh`)? > > > > > Thank you in advance. > > > > > On May 15, 10:26 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > > > > Actually, the idf is computed per shard (by default). You can use a > > > > search > > > > type set to dfs\_query\_then\_fetch, which will add another phase to teh > > > > search to compute distributed frequencies (but note it wil be > > > > slower). > > > > > > On Sun, May 13, 2012 at 8:40 PM, Crwe [tester.teste...@gmail.com](mailto:tester.teste...@gmail.com) > > > > wrote: > > > > > > > > > On May 13, 2:06 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > > > > > > > > I don't really understand what is the problem you have with > > > > > > search > > > > > > across > > > > > > the two indices... > > > > > > > For one, the relevance scores from different indexes seem to be > > > > > uncomparable. > > > > > > > In particular, the idf values seem to be calculated for each index > > > > > separately. This means that when results from two indexes are > > > > > merged > > > > > into a single ranked list, it's really comparing apples to oranges. > > > > > > > > On Fri, May 11, 2012 at 4:39 AM, Kinesh Satiya \< > > > > > > [ksat...@mysocialcloud.com](mailto:ksat...@mysocialcloud.com)\>wrote: > > > > > > > > > Hi, > > > > > > > > > I have two indexes, they basically index notes. A note has a > > > > > > > noteid, > > > > > > > title description, userid. > > > > > > > > > So my first index is notes(index)/note(mapping) with noteid > > > > > > > > > Second is note content which has list of note ids and note > > > > > > > content > > > > > > > and > > > > > > > id for this is sha1(title) and list of userids who have this > > > > > > > note > > > > > > > in > > > > > > > common. So note title is common between these two indexes. > > > > > > > > > All I want to do is search across both of theses indexes, with > > > > > > > individual boost factors for title description and content > > > > > > > with a > > > > > > > given keyword. > > > > > > > > > I know how to do the individual searches with bool and adding > > > > > > > boost > > > > > > > factors. > > > > > > > Currently I first search on first index with the given keyword > > > > > > > and > > > > > > > then search on next index for content and then merge both the > > > > > > > search > > > > > > > results. > > > > > > > > > But I do want to search across both of these indexes together > > > > > > > so > > > > > > > that > > > > > > > I can have relevant order. > > > > > > > > > Thanks ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:27am UTC](https://discuss.elastic.co/t/search-across-multiple-indexes/7647/9 "2017-07-06T03:27:27Z")