# Using differents analysers based on the document language **URL:** \ **Category:** Elasticsearch\ **Created:** [May 2, 2013, 1:41pm UTC](https://discuss.elastic.co/t/using-differents-analysers-based-on-the-document-language/11779 "2013-05-02T13:41:34Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Lucas\_Corte\_Real\_Sal](https://avatars.discourse-cdn.com/v4/letter/l/f05b48/32.png) [@Lucas\_Corte\_Real\_Sal](https://discuss.elastic.co/u/Lucas_Corte_Real_Sal)\ **Post date:** [May 2, 2013, 1:41pm UTC](https://discuss.elastic.co/t/using-differents-analysers-based-on-the-document-language/11779/1 "2013-05-02T13:41:34Z")
Hi everyone! Sorry if this is a dumb question, but I can't find anything about this at the documentation or foruns on the web. In my office we are using ES to index crawled news from the web. We crawl news in three languages: english, portuguese and spanish. Each news is indexed in the same index (don't distinguished by language in the indexing process). We have a service that queries this index and retrieves some documents to the client. Here is my problem: I would like to choose which analyser to be used based on the document language, mainly because of the stopwords (intrinsic to the idiom). Here is a simple example. Imagine that my query is "Rio de Janeiro". So I have 3 tokens ("Rio", "de", "Janeiro"). For english and spanish there aren't stopwords in this query, but in portuguese the token "de" is a stopword. So, in the query, while evaluating documents, if the document language is portuguese, I want that ES considers the token "de" as a stopword, while if the document language is english or spanish don't. Doing some research, I figured out that I have 3 choices: 1- Index my documents separately by idiom, specifying for each one of the index which analyser that must be used. (HARD because the software is already in production); 2- Build an adapter between my service and the ES API. This adapter will split the original query in three queries, one for each language. Each one of these queries will be used to search for documents of it's respective language using the respective analyser; 3- Find some magic functionality that allows me to choose the correct analyser that should be used for each document, based on the document language (I don't think that this option exists). So, I am looking for guidance to which way I should chose. If anyone have another option please let me know. Best regards, Lucas Saldanha -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Caio\_D\_Angelo](https://avatars.discourse-cdn.com/v4/letter/c/5f8ce5/32.png) [@Caio\_D\_Angelo](https://discuss.elastic.co/u/Caio_D_Angelo)\ **Post date:** [May 2, 2013, 8:11pm UTC](https://discuss.elastic.co/t/using-differents-analysers-based-on-the-document-language/11779/2 "2013-05-02T20:11:23Z")
We dealed with something like that here. In short: Your first option was our final one. Although, our system wasn't production yet at that time. Your second option is a little complicated when it comes to pagination. We considered it, but not took it. Your third would be great, but we can't find the trick in time to deploy our solution. It turns us out later, that you can specify different analyzers on indexing, but we didn't spend much time trying this. Maybe you can go by this way, indexing with different analyzers and see what happens on searching. I surely would give it a try. Em quinta-feira, 2 de maio de 2013 10h41min34s UTC-3, Lucas Corte Real Saldanha escreveu: > Hi everyone! > > Sorry if this is a dumb question, but I can't find anything about this at > the documentation or foruns on the web. > > In my office we are using ES to index crawled news from the web. We crawl > news in three languages: english, portuguese and spanish. Each news is > indexed in the same index (don't distinguished by language in the indexing > process). > > We have a service that queries this index and retrieves some documents to > the client. Here is my problem: I would like to choose which analyser to be > used based on the document language, mainly because of the stopwords > (intrinsic to the idiom). > > Here is a simple example. Imagine that my query is "Rio de Janeiro". So I > have 3 tokens ("Rio", "de", "Janeiro"). For english and spanish there > aren't stopwords in this query, but in portuguese the token "de" is a > stopword. So, in the query, while evaluating documents, if the document > language is portuguese, I want that ES considers the token "de" as a > stopword, while if the document language is english or spanish don't. > > Doing some research, I figured out that I have 3 choices: > > 1- Index my documents separately by idiom, specifying for each one of the > index which analyser that must be used. (HARD because the software is > already in production); > > 2- Build an adapter between my service and the ES API. This adapter will > split the original query in three queries, one for each language. Each one > of these queries will be used to search for documents of it's respective > language using the respective analyser; > > 3- Find some magic functionality that allows me to choose the correct > analyser that should be used for each document, based on the document > language (I don't think that this option exists). > > So, I am looking for guidance to which way I should chose. If anyone have > another option please let me know. > > Best regards, > Lucas Saldanha -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:38am UTC](https://discuss.elastic.co/t/using-differents-analysers-based-on-the-document-language/11779/3 "2017-07-06T02:38:34Z")