# Indexing without knowing the language **URL:** \ **Category:** Elasticsearch\ **Created:** [December 17, 2011, 4:42pm UTC](https://discuss.elastic.co/t/indexing-without-knowing-the-language/6185 "2011-12-17T16:42:43Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![James\_Cook\_3](https://avatars.discourse-cdn.com/v4/letter/j/d2c977/32.png) [@James\_Cook\_3](https://discuss.elastic.co/u/James_Cook_3)\ **Post date:** [December 17, 2011, 4:42pm UTC](https://discuss.elastic.co/t/indexing-without-knowing-the-language/6185/1 "2011-12-17T16:42:43Z")
I have a use case where several different languages can be used in a forum. I don't have any indication which language is used by which users, and in some cases, multiple languages might be used in the same posts. My document includes the typical properties one might expect for a forum post along with the body of the message. What strategies might I employ so users can search for the german posts using german keywords, japanese posts using japanese keywords, etc? ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [December 18, 2011, 5:45am UTC](https://discuss.elastic.co/t/indexing-without-knowing-the-language/6185/2 "2011-12-18T05:45:53Z")
Hi James, Sounds like you should simply identify the language(s) of documents before indexing them and then analyze them appropriately. There is a language identifier that's included in Solr 3.5.0 you could rip out. Otis On Dec 17, 11:42 am, James Cook [jc...@pykl.com](mailto:jc...@pykl.com) wrote: > I have a use case where several different languages can be used in a forum. > I don't have any indication which language is used by which users, and in > some cases, multiple languages might be used in the same posts. > > My document includes the typical properties one might expect for a forum > post along with the body of the message. > > What strategies might I employ so users can search for the german posts > using german keywords, japanese posts using japanese keywords, etc? ---
**Author:** ![karmi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karmi/32/44951_2.png) [@karmi](https://discuss.elastic.co/u/karmi)\ **Post date:** [December 18, 2011, 8:47am UTC](https://discuss.elastic.co/t/indexing-without-knowing-the-language/6185/3 "2011-12-18T08:47:30Z")
Hi James, there's also a Google "Compact Language Detector", [GitHub - mikemccand/chromium-compact-language-detector: Automatically exported from code.google.com/p/chromium-compact-language-detector](http://code.google.com/p/chromium-compact-language-detector/), which seemed very nice to me, with easy wrapping from other languages (eg. [Changing Bits: Language detection with Google's Compact Language Detector](http://blog.mikemccandless.com/2011/10/language-detection-with-googles-compact.html), [GitHub - jtoy/cld: compact language detection in ruby](https://github.com/jtoy/cld)). Detecting the language won't be the hard part here, I think. The hard part is defining proper mapping for those languages (multifields?, different properties?, etc). I think multifields should work really well here, but I don't know of any strategy which would allow magically analyzing queries with german analyzers and searching german fields, and vice versa for english/japanese, etc., without the user explicitely setting the language, or "trying out" the search against all the multi fields and depending on the \_score to sort them right... This does not sound so "simple" as Otis' response suggests :), so I may be missing something. Karel On Dec 17, 5:42 pm, James Cook [jc...@pykl.com](mailto:jc...@pykl.com) wrote: > I have a use case where several different languages can be used in a forum. > I don't have any indication which language is used by which users, and in > some cases, multiple languages might be used in the same posts. > > My document includes the typical properties one might expect for a forum > post along with the body of the message. > > What strategies might I employ so users can search for the german posts > using german keywords, japanese posts using japanese keywords, etc? ---
**Author:** ![James\_Cook\_3](https://avatars.discourse-cdn.com/v4/letter/j/d2c977/32.png) [@James\_Cook\_3](https://discuss.elastic.co/u/James_Cook_3)\ **Post date:** [December 19, 2011, 1:04am UTC](https://discuss.elastic.co/t/indexing-without-knowing-the-language/6185/4 "2011-12-19T01:04:54Z")
I wonder if a "smart" analyzer could be employed at both index and search time? At index time, the smart analyzer would introduce a new multitype to the mapping for the detected language, and at search time, the smart analyzer would do something similar? ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [December 19, 2011, 5:34am UTC](https://discuss.elastic.co/t/indexing-without-knowing-the-language/6185/5 "2011-12-19T05:34:54Z")
Hi, On Dec 18, 3:47 am, karmi [karel.mina...@gmail.com](mailto:karel.mina...@gmail.com) wrote: > Hi James, > > there's also a Google "Compact Language Detector",[GitHub - mikemccand/chromium-compact-language-detector: Automatically exported from code.google.com/p/chromium-compact-language-detector](http://code.google.com/p/chromium-compact-language-detector/), which > seemed very nice to me, with easy wrapping from other languages (eg.[http://blog.mikemccandless.com/2011/10/language-detection-with-google](http://blog.mikemccandless.com/2011/10/language-detection-with-google)...,[GitHub - jtoy/cld: compact language detection in ruby](https://github.com/jtoy/cld)). > > Detecting the language won't be the hard part here, I think. The hard > part is defining proper mapping for those languages (multifields?, > different properties?, etc). I think multifields should work really > well here, but I don't know of any strategy which would allow > magically analyzing queries with german analyzers and searching german > fields, and vice versa for english/japanese, etc., without the user Are you referring to? > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... If (don't know) this can indeed be used for multilingual situations like this one, it would be great for ES documentation to include that example, because dealing with this sort of problem will become a FAQ if it's not already. ## Otis Sematext is Hiring World-Wide -- [Jobs](http://sematext.com/about/jobs.html) > explicitely setting the language, or "trying out" the search against > all the multi fields and depending on the \_score to sort them right... > > This does not sound so "simple" as Otis' response suggests :), so I > may be missing something. > > Karel > > On Dec 17, 5:42 pm, James Cook [jc...@pykl.com](mailto:jc...@pykl.com) wrote: > > > I have a use case where several different languages can be used in a forum. > > I don't have any indication which language is used by which users, and in > > some cases, multiple languages might be used in the same posts. > > > My document includes the typical properties one might expect for a forum > > post along with the body of the message. > > > What strategies might I employ so users can search for the german posts > > using german keywords, japanese posts using japanese keywords, etc? ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [December 19, 2011, 5:37am UTC](https://discuss.elastic.co/t/indexing-without-knowing-the-language/6185/6 "2011-12-19T05:37:22Z")
Hi, On Dec 18, 8:04 pm, James Cook [jc...@pykl.com](mailto:jc...@pykl.com) wrote: > I wonder if a "smart" analyzer could be employed at both index and search > time? > > At index time, the smart analyzer would introduce a new multitype to the > mapping for the detected language, and at search time, the smart analyzer > would do something similar? I'm all eyeballs in this thread! 🙂 Re search-time language identification, the common problem is the shortness of query strings ☹ So in my experience one tries to figure out the user's language some other way (explicit selection of from preferences or...) ## Otis Sematext is Hiring World-Wide -- [Jobs](http://sematext.com/about/jobs.html) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:45am UTC](https://discuss.elastic.co/t/indexing-without-knowing-the-language/6185/7 "2017-07-06T03:45:00Z")