Mult-language searchable in one field

Martin_3 · May 9, 2014, 10:40am

Hi,

I've got descriptions in different languages with different analyzers like
this:

description_lang: {

properties: {
- de: {
  - analyzer: my_analyzer_de
  - type: string
    }
- en: {
  - analyzer: my_analyzer_en
  - type: string
    }
    }

}

They are searchabel with "description_lang.de" and "description_lang.en"but I'd like to search them all together with only
"
description_lang" is that possible somehow ?

Thank you very much,
Martin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b1edecec-4722-4e3f-b82c-eb742d7cb3ad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Martin_3 · May 9, 2014, 10:45am

and I cant' use the "_all" field because of own scoring.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5d87fc34-f433-4b2f-83b9-7a0118b1ab8d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Itamar_Syn_Hershko · May 9, 2014, 10:45am

And then what analyzer you will use for that?

It is doable, but I'd strongly suggest against it unless you know what you
are doing:

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Fri, May 9, 2014 at 1:40 PM, Martin martin.konradi@echtzeiten.comwrote:

Hi,

I've got descriptions in different languages with different analyzers like
this:

description_lang: {

properties: {

de: {

analyzer: my_analyzer_de

type: string
}

en: {

analyzer: my_analyzer_en

type: string
}
}

}

They are searchabel with "description_lang.de" and "description_lang.en"but I'd like to search them all together with only
"
description_lang" is that possible somehow ?

Thank you very much,
Martin

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b1edecec-4722-4e3f-b82c-eb742d7cb3ad%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/b1edecec-4722-4e3f-b82c-eb742d7cb3ad%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtryLtndSjLdZu8dnmARMpsZhzxHc1y18XFKxUtv_DzYQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Martin_3 · May 9, 2014, 12:46pm

Thank you - I thought may be there is another type like object-type which
would provide it.
Actually it should be a type like the array-type but with fields you could
analyse separately.
There would be different analysers for each language (especially because of
stemming).

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a8aa260b-a91b-4300-877a-c69928bdcfc8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Itamar_Syn_Hershko · May 9, 2014, 12:52pm

You can do that, but by generating the query on the client side. I'd then
use a Boolean query wrapping a Match query, each on a different language
field and specifying the analyzer explicitly.

I think that's a healthier approach as it's easily changable and easier to
debug, as often times multi-lingual search demands a lot of attention

Not long ago I gave a talk about this topic, you might find it helpful:

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Fri, May 9, 2014 at 3:46 PM, Martin martin.konradi@echtzeiten.comwrote:

Thank you - I thought may be there is another type like object-type which
would provide it.
Actually it should be a type like the array-type but with fields you could
analyse separately.
There would be different analysers for each language (especially because
of stemming).

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a8aa260b-a91b-4300-877a-c69928bdcfc8%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/a8aa260b-a91b-4300-877a-c69928bdcfc8%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtaEHYq8ELY-A0mTnLRkvCk76GXT7KeDqQfYva8by4K8g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Martin_3 · May 9, 2014, 1:08pm

OK - I thought I could leave all my queries and add an additional filter if
the item is available in that language.
I'll watch your talk!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fcd4b3b2-a254-4f0d-9f44-f4174e303a24%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jprante · May 9, 2014, 1:15pm

You can index two fields into a third common field with "index_name" in
combination with "multi_field". This works like a custom "_all" field.

Jörg

On Fri, May 9, 2014 at 3:08 PM, Martin martin.konradi@echtzeiten.comwrote:

OK - I thought I could leave all my queries and add an additional filter
if the item is available in that language.
I'll watch your talk!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/fcd4b3b2-a254-4f0d-9f44-f4174e303a24%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/fcd4b3b2-a254-4f0d-9f44-f4174e303a24%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEC018Hmf%3DjXhTOoYtX5WE8255hwWcLjb1QtmbnWr-twQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Martin_3 · May 9, 2014, 2:19pm

ah wow - that's cool. I just came across your
http://jprante.github.io/lessons/2012/05/11/ISBN-search-with-Elasticsearch.html
which explains it in detail.
That's pretty close to what I want but I just found a new problem: There
should be a highlighting of the found results - so I guess I'll have to go
with multiple boolean should queries.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7d18189b-9950-4020-bf3d-dd26680cfa46%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jprante · May 9, 2014, 9:31pm

Elasticsearch has improved ithe "index_name" with the "copy_to" mechanism.

I have set up a gist to demonstrate multilingual highlighting, maybe it is
useful.

gist.github.com

https://gist.github.com/jprante/9f7523bb59ea654a3931

highlighting-copy-to.sh


curl -XDELETE 'localhost:9200/test'

curl -XPUT 'localhost:9200/test/' -d '
{
  "mappings" : {
       "_default_": {
           "properties" : {
               "name" : {
                    "type" : "string",

This file has been truncated. show original

Jörg

On Fri, May 9, 2014 at 4:19 PM, Martin martin.konradi@echtzeiten.comwrote:

ah wow - that's cool. I just came across your
http://jprante.github.io/lessons/2012/05/11/ISBN-search-with-Elasticsearch.htmlwhich explains it in detail.
That's pretty close to what I want but I just found a new problem: There
should be a highlighting of the found results - so I guess I'll have to go
with multiple boolean should queries.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7d18189b-9950-4020-bf3d-dd26680cfa46%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/7d18189b-9950-4020-bf3d-dd26680cfa46%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHEzVWVJBkjOV2sLoWczdo2jSvaftmN7yFL4Xvjk7wN4w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.