# Multiple analyzers in "\_all" field? **URL:** \ **Category:** Elasticsearch\ **Created:** [November 8, 2013, 3:40pm UTC](https://discuss.elastic.co/t/multiple-analyzers-in--all-field/14324 "2013-11-08T15:40:19Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Pawel\_Mlynarczyk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pawel_mlynarczyk/32/1568_2.png) [@Pawel\_Mlynarczyk](https://discuss.elastic.co/u/Pawel_Mlynarczyk)\ **Post date:** [November 8, 2013, 3:40pm UTC](https://discuss.elastic.co/t/multiple-analyzers-in--all-field/14324/1 "2013-11-08T15:40:19Z")
Hi I've got a multilingual documents to index. I want to create a full text search, so the first thing on my mind was to use string query with the \_all field. The problem is that the \_all field has it's own analyzer, so the fields specific analyzers are not used (data is not analyzed properly). Is there a way to use field's aproppriate analyzer when copying data to \_all instead of just reanalyzing it with the \_all's analyzer? Creating separate index for each language is not a good solution for my case, because I've got milions of documents and every one of them contains fields in more than one language + a number of language independent fields. That means I would end up having heavily duplicated data in every index. Thanks in advance Paweł Młynarczyk -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 8, 2013, 4:51pm UTC](https://discuss.elastic.co/t/multiple-analyzers-in--all-field/14324/2 "2013-11-08T16:51:41Z")
I would not use \_all field for that but I would probably disable it and the use multifield type on your "content" field. Probably one sub field per language. See: [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-all-field.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-all-field.html) [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-multi-field-type.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-multi-field-type.html) HTH -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr 8 novembre 2013 at 16:40:23, Paweł Młynarczyk ([zwarios@gmail.com](mailto:zwarios@gmail.com)) a écrit: Hi I've got a multilingual documents to index. I want to create a full text search, so the first thing on my mind was to use string query with the \_all field. The problem is that the \_all field has it's own analyzer, so the fields specific analyzers are not used (data is not analyzed properly). Is there a way to use field's aproppriate analyzer when copying data to \_all instead of just reanalyzing it with the \_all's analyzer? Creating separate index for each language is not a good solution for my case, because I've got milions of documents and every one of them contains fields in more than one language + a number of language independent fields. That means I would end up having heavily duplicated data in every index. Thanks in advance ## Paweł Młynarczyk You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Pawel\_Mlynarczyk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pawel_mlynarczyk/32/1568_2.png) [@Pawel\_Mlynarczyk](https://discuss.elastic.co/u/Pawel_Mlynarczyk)\ **Post date:** [November 8, 2013, 6:22pm UTC](https://discuss.elastic.co/t/multiple-analyzers-in--all-field/14324/3 "2013-11-08T18:22:50Z")
Thanks for your reply. You suggest creating a multi field equivalent of the '\_all' field, but isn't that a waste to analyze all the language dependant data with every analyzer? I mean if I would create that kind of custom '\_all' field and put there aggregated data from all the language dependant fields, than I would end up having X '\_all' fields (where X is the number of languages) right? Additionaly would I have any option to boost a particular, more important field? (In my case, every language have more than 1 field and some of them are more important) Paweł Młynarczyk W dniu piątek, 8 listopada 2013 17:51:41 UTC+1 użytkownik David Pilato napisał: > I would not use \_all field for that but I would probably disable it and > the use multifield type on your "content" field. > Probably one sub field per language. > > See: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-all-field.html) > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-multi-field-type.html) > > HTH > > -- > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > > 8 novembre 2013 at 16:40:23, Paweł Młynarczyk ([zwa...@gmail.com](mailto:zwa...@gmail.com)\) > a écrit: > > Hi > > I've got a multilingual documents to index. I want to create a full text > search, so the first thing on my mind was to use string query with the \_all > field. The problem is that the \_all field has it's own analyzer, so the > fields specific analyzers are not used (data is not analyzed properly). Is > there a way to use field's aproppriate analyzer when copying data to \_all > instead of just reanalyzing it with the \_all's analyzer? > > Creating separate index for each language is not a good solution for my > case, because I've got milions of documents and every one of them contains > fields in more than one language + a number of language independent fields. > That means I would end up having heavily duplicated data in every index. > > Thanks in advance > > ## Paweł Młynarczyk > > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 8, 2013, 6:37pm UTC](https://discuss.elastic.co/t/multiple-analyzers-in--all-field/14324/4 "2013-11-08T18:37:44Z")
I don't understand. What can you do? There are two options for me: - you know the language in advance (because you can detect it or whatever) and in that case, use \_analyzer field to extract the analyzer to apply from the json document itself: [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-analyzer-field.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-analyzer-field.html) - you don't know the language. For that last item, how can you proceed? Random analyzer? Try more than one analyzer? No analyzer at all (yes you may want to use ngrams to try to make it work but I guess that it will generate a lot of false positive results). More than one analyzer is multifield. You're right. Behind the scene, it's like having X fields one for each language. But instead of providing a json document like: { "content\_fr":"mon contenu francais", "content\_en":"mon contenu francais", "content\_de":"mon contenu francais" } You will be able to provide { "content":"mon contenu francais", } So in term of \_source storage, you won't pay the price 3 time. In term of inverted index, yes you will consume space for content.fr, content.en and content.de. Makes sense? -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr 8 novembre 2013 at 19:22:55, Paweł Młynarczyk ([zwarios@gmail.com](mailto:zwarios@gmail.com)) a écrit: Thanks for your reply. You suggest creating a multi field equivalent of the '\_all' field, but isn't that a waste to analyze all the language dependant data with every analyzer? I mean if I would create that kind of custom '\_all' field and put there aggregated data from all the language dependant fields, than I would end up having X '\_all' fields (where X is the number of languages) right? Additionaly would I have any option to boost a particular, more important field? (In my case, every language have more than 1 field and some of them are more important) Pawe³ M³ynarczyk W dniu pi±tek, 8 listopada 2013 17:51:41 UTC+1 u¿ytkownik David Pilato napisa³: I would not use \_all field for that but I would probably disable it and the use multifield type on your "content" field. Probably one sub field per language. See: [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-all-field.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-all-field.html) [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-multi-field-type.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-multi-field-type.html) HTH -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr 8 novembre 2013 at 16:40:23, Pawe³ M³ynarczyk ([zwa...@gmail.com](mailto:zwa...@gmail.com)) a écrit: Hi I've got a multilingual documents to index. I want to create a full text search, so the first thing on my mind was to use string query with the \_all field. The problem is that the \_all field has it's own analyzer, so the fields specific analyzers are not used (data is not analyzed properly). Is there a way to use field's aproppriate analyzer when copying data to \_all instead of just reanalyzing it with the \_all's analyzer? Creating separate index for each language is not a good solution for my case, because I've got milions of documents and every one of them contains fields in more than one language + a number of language independent fields. That means I would end up having heavily duplicated data in every index. Thanks in advance ## Pawe³ M³ynarczyk ## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Pawel\_Mlynarczyk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pawel_mlynarczyk/32/1568_2.png) [@Pawel\_Mlynarczyk](https://discuss.elastic.co/u/Pawel_Mlynarczyk)\ **Post date:** [November 10, 2013, 11:15am UTC](https://discuss.elastic.co/t/multiple-analyzers-in--all-field/14324/5 "2013-11-10T11:15:52Z")
Yes it does make sense. The thing is, I've got a doc that has fields: name.en, [name.fr](http://name.fr), [name.de](http://name.de), name.pl, description.en, [description.fr](http://description.fr), [description.de](http://description.de), description.pl, etc - a lot of others, language independent. Creating the 'content' multi field would aggregate data from multiple languages in each document - that would cause the index to be huge wouldn't it? Additionally, I'd have no option to boost a particular field (name in this case) to be more important within the search. I've already tried couple of solutions, but every one of them has some drawbacks: 1. Using the '\_all' field - in this case I can't use language specific analyzers. 2. Using separate index for every language - in this case the language independent data of each doc have to be indexed multiple times. 3. Using 'fields' attribute of the 'string query' to do a text query on the right language fields. This seems a good one, but since I've got a lot of fields to execute this query against, then the query performance drops down as compared to query against the '\_all' field. 4. Aggregating language dependant data to one field, and execute a string query against that field. In this case I cannot boost a particular field. I am looking to minimise the drawbacks. Any other ideas? W dniu piątek, 8 listopada 2013 19:37:44 UTC+1 użytkownik David Pilato napisał: > I don't understand. What can you do? > > There are two options for me: > > - you know the language in advance (because you can detect it or whatever) > and in that case, use \_analyzer field to extract the analyzer to apply from > the json document itself: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-analyzer-field.html) > - you don't know the language. > > For that last item, how can you proceed? Random analyzer? Try more than > one analyzer? No analyzer at all (yes you may want to use ngrams to try to > make it work but I guess that it will generate a lot of false positive > results). > More than one analyzer is multifield. You're right. Behind the scene, it's > like having X fields one for each language. But instead of providing a json > document like: > > { > "content\_fr":"mon contenu francais", > "content\_en":"mon contenu francais", > "content\_de":"mon contenu francais" > } > > You will be able to provide > { > "content":"mon contenu francais", > } > > So in term of \_source storage, you won't pay the price 3 time. In term of > inverted index, yes you will consume space for [content.fr](http://content.fr), content.en and > [content.de](http://content.de). > > Makes sense? > > -- > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > > 8 novembre 2013 at 19:22:55, Paweł Młynarczyk ([zwa...@gmail.com](mailto:zwa...@gmail.com)\) > a écrit: > > Thanks for your reply. > > You suggest creating a multi field equivalent of the '\_all' field, but > isn't that a waste to analyze all the language dependant data with every > analyzer? I mean if I would create that kind of custom '\_all' field and put > there aggregated data from all the language dependant fields, than I would > end up having X '\_all' fields (where X is the number of languages) right? > Additionaly would I have any option to boost a particular, more important > field? (In my case, every language have more than 1 field and some of them > are more important) > > Pawe³ M³ynarczyk > > W dniu pi±tek, 8 listopada 2013 17:51:41 UTC+1 u¿ytkownik David Pilato > napisa³: > > > I would not use \_all field for that but I would probably disable it and > > the use multifield type on your "content" field. > > Probably one sub field per language. > > > > See: > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-all-field.html) > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-multi-field-type.html) > > > > HTH > > > > ``` > > -- > > > > ``` > > > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > > > > 8 novembre 2013 at 16:40:23, Pawe³ M³ynarczyk ([zwa...@gmail.com](mailto:zwa...@gmail.com)) a écrit: > > > > Hi > > > > I've got a multilingual documents to index. I want to create a full text > > search, so the first thing on my mind was to use string query with the \_all > > field. The problem is that the \_all field has it's own analyzer, so the > > fields specific analyzers are not used (data is not analyzed properly). Is > > there a way to use field's aproppriate analyzer when copying data to \_all > > instead of just reanalyzing it with the \_all's analyzer? > > > > Creating separate index for each language is not a good solution for my > > case, because I've got milions of documents and every one of them contains > > fields in more than one language + a number of language independent fields. > > That means I would end up having heavily duplicated data in every index. > > > > Thanks in advance > > > > ## Pawe³ M³ynarczyk > > > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 11, 2013, 9:05am UTC](https://discuss.elastic.co/t/multiple-analyzers-in--all-field/14324/6 "2013-11-11T09:05:21Z")
You can use the \_all field with the combo analyzer > **[yakaz/elasticsearch-analysis-combo](https://github.com/yakaz/elasticsearch-analysis-combo)** > > elasticsearch-analysis-combo - Elasticsearch Combo Analyzer It can concatenate many tokens from many analyzers into one field. Note, that scoring is a little bit skewed. Do not forget to use the unique token filter to reduce multiple occurances of tokens [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-unique-tokenfilter.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-unique-tokenfilter.html) Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:07am UTC](https://discuss.elastic.co/t/multiple-analyzers-in--all-field/14324/7 "2017-07-06T02:07:54Z")