# Searching on \_all vs bool query on a (large) number of fields **URL:** \ **Category:** Elasticsearch\ **Created:** [September 25, 2014, 10:08am UTC](https://discuss.elastic.co/t/searching-on--all-vs-bool-query-on-a-large-number-of-fields/19959 "2014-09-25T10:08:44Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![mooky](https://avatars.discourse-cdn.com/v4/letter/m/43a26b/32.png) [@mooky](https://discuss.elastic.co/u/mooky)\ **Post date:** [September 25, 2014, 10:08am UTC](https://discuss.elastic.co/t/searching-on--all-vs-bool-query-on-a-large-number-of-fields/19959/1 "2014-09-25T10:08:44Z")
The \_all field is super-convenient for doing a google-style search across all fields of a document. The mapping makes it easy to include/exclude fields from \_all in a declarative fashion - including boost levels etc. It is a pattern I have been using for many years - by hand using lucene (before elastic), using zzz\_all (or whatever it was called) in Compass and it has served me an my users well. However, one significant shortcoming is that you cannot do any highlighting. (And there are probably some other shortcomings I am not aware of...) I read in the documentation: [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/search-lite.html) TIP:The \_all field is a useful feature while you are getting started with a > new application. Later, you will find that you have more control over your > search results if you query specific fields instead of the \_all field. > When the \_all field is no longer useful to you, you can disable it, as > explained in Metadata: \_all field > [http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/root-object.html#all-field](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/root-object.html#all-field) > . And this lead me to re-thinking how it might be done. I can turn the problem inside-out. Instead of using \_all, I could instead create a more complicated query across all of the fields that are included in \_all. I can dynamically generate the query from my field metadata (an enum). My question mark is what kind of effect that will have on performance given I am indexing structured data (not text, per se). To give an example, one of my documents has 115 (!) fields. So my query will go from: { "query" : { "prefix" : { "\_all" : { "prefix" : "moo" } } } } To { "bool" : { "should" : [ {"prefix" : { "field1" : "moo" }}, {"prefix" : { "field2" : "moo" }}, {"prefix" : { "field3" : "moo" }}, {"prefix" : { "field4" : "moo" }}, {"prefix" : { "field5" : "moo" }}, {"prefix" : { "field6" : "moo" }}, . . . {"prefix" : { "field114" : "moo" }}, {"prefix" : { "field115" : "moo" }}, ] } } Ouch. No? What kind of performance impact will this have? Diabolical? The upside is that I now have the flexibility to get some meaningful highlighting .... but at what cost? Anyone have experience with this? Cheers, -N -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/512274e5-249f-482c-bf9c-27baa879a253%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/512274e5-249f-482c-bf9c-27baa879a253%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![brian\_yoder](https://avatars.discourse-cdn.com/v4/letter/b/f1d935/32.png) [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)\ **Post date:** [October 1, 2014, 4:14pm UTC](https://discuss.elastic.co/t/searching-on--all-vs-bool-query-on-a-large-number-of-fields/19959/2 "2014-10-01T16:14:49Z")
I would not say "Diabolical". Perhaps not optimal based on Lucene's internal design. But I do something similar with table-based synonyms. In other words, when matching a synonym of a word, I do not pre-build the database index with synonyms. Instead, I maintain a table (index/type) of words and their synonyms, query that table, retrieve the synonyms, and then create the second and final query that basically does an OR search across the word and its synonyms. (It's basically a group of should clauses, just like yours). I find that performance is fine. And accuracy and usefulness is superior. For example, a user query for synonym of the wild-carded BIG\* might find BIG, LARGE, HUGE and also BIGHORN, SHEEP. And so on; some of the synonym lists are rather long and with multiple words there are many should terms in the final query. And even with the multiple queries (first to resolve the synonyms, and the second to OR across them), performance is remarkably fast. It might be pushing Lucene a little, but I like the improved accuracy, and the ability to easily and regularly modify my synonym lists without any need to rebuild the hundreds of millions of documents that I am querying. So for your question, my suggestion is to go for it and it should perform well enough. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4d6ba249-c8b6-4870-af96-ed71ee1b2f7e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4d6ba249-c8b6-4870-af96-ed71ee1b2f7e%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Mark\_Harwood\_2](https://avatars.discourse-cdn.com/v4/letter/m/c77e96/32.png) [@Mark\_Harwood\_2](https://discuss.elastic.co/u/Mark_Harwood_2)\ **Post date:** [October 2, 2014, 9:06am UTC](https://discuss.elastic.co/t/searching-on--all-vs-bool-query-on-a-large-number-of-fields/19959/3 "2014-10-02T09:06:39Z")
> one significant shortcoming is that you cannot do any highlighting. Not necessarily true - see this feature which is primarily for the use case of searching on an "all" type field but highlighting results using detailed fields: [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-highlighting.html#field-match) Bear in mind that when searching across multiple fields Lucene's natural tendency to favour rare terms will do odd things like ranking the most bizarre interpretation of field choice e.g. firstName:Smith ranks higher than lastName:Smith. The multi-match query has some special features built into the cross-field matching modes to counter-act this tendency. [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-multi-match-query.html#type-cross-fields) Cheers Mark On Thursday, September 25, 2014 11:08:44 AM UTC+1, mooky wrote: > The \_all field is super-convenient for doing a google-style search across > all fields of a document. The mapping makes it easy to include/exclude > fields from \_all in a declarative fashion - including boost levels etc. > It is a pattern I have been using for many years - by hand using lucene > (before elastic), using zzz\_all (or whatever it was called) in Compass and > it has served me an my users well. > > However, one significant shortcoming is that you cannot do any > highlighting. (And there are probably some other shortcomings I am not > aware of...) > > I read in the documentation: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/search-lite.html) > > TIP:The \_all field is a useful feature while you are getting started with > > > a new application. Later, you will find that you have more control over > > your search results if you query specific fields instead of the \_all field. > > When the \_all field is no longer useful to you, you can disable it, as > > explained in Metadata: \_all field > > [http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/root-object.html#all-field](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/root-object.html#all-field) > > . > > And this lead me to re-thinking how it might be done. > > I can turn the problem inside-out. Instead of using \_all, I could instead > create a more complicated query across all of the fields that are included > in \_all. I can dynamically generate the query from my field metadata (an > enum). > > My question mark is what kind of effect that will have on performance > given I am indexing structured data (not text, per se). > To give an example, one of my documents has 115 (!) fields. > > So my query will go from: > > { > "query" : { > "prefix" : { > "\_all" : { > "prefix" : "moo" > } > } > } > } > > To > > { > "bool" : { > "should" : [ > {"prefix" : { "field1" : "moo" }}, > {"prefix" : { "field2" : "moo" }}, > {"prefix" : { "field3" : "moo" }}, > {"prefix" : { "field4" : "moo" }}, > {"prefix" : { "field5" : "moo" }}, > {"prefix" : { "field6" : "moo" }}, > . > . > . > {"prefix" : { "field114" : "moo" }}, > {"prefix" : { "field115" : "moo" }}, > ] > } > } > > Ouch. No? > What kind of performance impact will this have? Diabolical? > > The upside is that I now have the flexibility to get some meaningful > highlighting .... but at what cost? > Anyone have experience with this? > > Cheers, > -N -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/cc4ccc93-0d69-4f64-b7d0-db878d0c0b9b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cc4ccc93-0d69-4f64-b7d0-db878d0c0b9b%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![mooky](https://avatars.discourse-cdn.com/v4/letter/m/43a26b/32.png) [@mooky](https://discuss.elastic.co/u/mooky)\ **Post date:** [October 8, 2014, 11:40am UTC](https://discuss.elastic.co/t/searching-on--all-vs-bool-query-on-a-large-number-of-fields/19959/4 "2014-10-08T11:40:39Z")
> > > > one significant shortcoming is that you cannot do any highlighting. > > > > Not necessarily true - see this feature which is primarily for the use > > > > case of searching on an "all" type field but highlighting results using > > > > detailed fields: > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. Hm ok. That looks interesting, but what exactly do I have to do? I cant seem to get any highlighting ... I added highlighted field "\*" (ie all of them - but just to be sure I added just 1 known field with no wildcarding and queried based on a value within it). I set require\_field\_match to false (and to true, just in case I misunderstood) I set force\_source to true (because I am not storing any fields). But I get no highlighting ... what have I missed? > > The multi-match query has some special features built into the > > cross-field matching modes to counter-act this tendency. > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. Ah. Multi-match looks much better than constructing a big bool query. Thanks for the pointer. Thanks. -N On Thursday, 2 October 2014 10:06:39 UTC+1, Mark Harwood wrote: > > one significant shortcoming is that you cannot do any highlighting. > > Not necessarily true - see this feature which is primarily for the use > case of searching on an "all" type field but highlighting results using > detailed fields: > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-highlighting.html#field-match) > > Bear in mind that when searching across multiple fields Lucene's natural > tendency to favour rare terms will do odd things like ranking the most > bizarre interpretation of field choice e.g. firstName:Smith ranks higher > than lastName:Smith. > The multi-match query has some special features built into the cross-field > matching modes to counter-act this tendency. > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-multi-match-query.html#type-cross-fields) > > Cheers > Mark > > On Thursday, September 25, 2014 11:08:44 AM UTC+1, mooky wrote: > > > The \_all field is super-convenient for doing a google-style search across > > all fields of a document. The mapping makes it easy to include/exclude > > fields from \_all in a declarative fashion - including boost levels etc. > > It is a pattern I have been using for many years - by hand using lucene > > (before elastic), using zzz\_all (or whatever it was called) in Compass and > > it has served me an my users well. > > > > However, one significant shortcoming is that you cannot do any > > highlighting. (And there are probably some other shortcomings I am not > > aware of...) > > > > I read in the documentation: > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/search-lite.html) > > > > TIP:The \_all field is a useful feature while you are getting started > > > > > with a new application. Later, you will find that you have more control > > > over your search results if you query specific fields instead of the > > > \_all field. When the \_all field is no longer useful to you, you can > > > disable it, as explained in Metadata: \_all field > > > [http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/root-object.html#all-field](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/root-object.html#all-field) > > > . > > > > And this lead me to re-thinking how it might be done. > > > > I can turn the problem inside-out. Instead of using \_all, I could instead > > create a more complicated query across all of the fields that are included > > in \_all. I can dynamically generate the query from my field metadata (an > > enum). > > > > My question mark is what kind of effect that will have on performance > > given I am indexing structured data (not text, per se). > > To give an example, one of my documents has 115 (!) fields. > > > > So my query will go from: > > > > { > > "query" : { > > "prefix" : { > > "\_all" : { > > "prefix" : "moo" > > } > > } > > } > > } > > > > To > > > > { > > "bool" : { > > "should" : [ > > {"prefix" : { "field1" : "moo" }}, > > {"prefix" : { "field2" : "moo" }}, > > {"prefix" : { "field3" : "moo" }}, > > {"prefix" : { "field4" : "moo" }}, > > {"prefix" : { "field5" : "moo" }}, > > {"prefix" : { "field6" : "moo" }}, > > . > > . > > . > > {"prefix" : { "field114" : "moo" }}, > > {"prefix" : { "field115" : "moo" }}, > > ] > > } > > } > > > > Ouch. No? > > What kind of performance impact will this have? Diabolical? > > > > The upside is that I now have the flexibility to get some meaningful > > highlighting .... but at what cost? > > Anyone have experience with this? > > > > Cheers, > > -N -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e7eb88cc-f463-448c-9d4c-70028b60753f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e7eb88cc-f463-448c-9d4c-70028b60753f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:57am UTC](https://discuss.elastic.co/t/searching-on--all-vs-bool-query-on-a-large-number-of-fields/19959/5 "2017-07-06T00:57:41Z")