Multi-field is not working


(Bernd Fehling) #1

I wonder why multi-field is not working.

At first, ES is always setting "include_in_all" to false by its own. The ES
guide for Multi Field Type isn't documenting this.
Put mapping is:
"myname" : {
"type" : "multi_field"
"fields" : {
"myname" : { "type" : "string" },
"yourname" : { "type" : "string" }
}
}

Get mapping shows:
"myname" : {
"type" : "multi_field"
"fields" : {
"myname" : { "type" : "string" },
"yourname" : { "type" : "string", "include_in_all" : false }
}
}

If I set "include_in_all" to true and send my setting to ES it responds
with {"ok":true,"acknowledged":true}.
No error message but also not setting "include_in_all" to true.

Why is it setting my multi-field fix to "include_in_all" : false?

What is the sense of a system not accepting my settings but also not
reporting any errors of other messages?

Second, I play aroung with this wrong setting and load some data.
A query to "curl -XGET localhost:9200/_search?q=*" shows that there is no
field "yourname".
With help of luke I analyze the index which shows the fields "myname" and
"myname.yourname" in the index and they also have the correct content.

Why is it not showing up when I query the index with ES?

Bernd


(David Pilato) #2

If I understood your last question, you want to find "yourname" in "_source"?
Source is exactly the document you sent.

And you sent only myname.

Is that what you were talking about?

David

--

Le 10 août 2012 à 08:51, Bernd Fehling bernd.fehling@googlemail.com a écrit :

I wonder why multi-field is not working.

At first, ES is always setting "include_in_all" to false by its own. The ES guide for Multi Field Type isn't documenting this.
Put mapping is:
"myname" : {
"type" : "multi_field"
"fields" : {
"myname" : { "type" : "string" },
"yourname" : { "type" : "string" }
}
}

Get mapping shows:
"myname" : {
"type" : "multi_field"
"fields" : {
"myname" : { "type" : "string" },
"yourname" : { "type" : "string", "include_in_all" : false }
}
}

If I set "include_in_all" to true and send my setting to ES it responds with {"ok":true,"acknowledged":true}.
No error message but also not setting "include_in_all" to true.

Why is it setting my multi-field fix to "include_in_all" : false?

What is the sense of a system not accepting my settings but also not reporting any errors of other messages?

Second, I play aroung with this wrong setting and load some data.
A query to "curl -XGET localhost:9200/_search?q=*" shows that there is no field "yourname".
With help of luke I analyze the index which shows the fields "myname" and "myname.yourname" in the index and they also have the correct content.

Why is it not showing up when I query the index with ES?

Bernd


(Bernd Fehling) #3

Actually I don't care where to find it, I only want to have it work like
copyField in Solr.
Load it into ES with one field e.g. "myname" but have it in the index in
"myname" and "yourname" or if nessessary in "myname.yourname".
Exactly as it is in Solr.
From all the mailings and sparse docs I found it was stated that this can
be done with "Multi Field Type".
Or not?


(David Pilato) #4

I don't know Solr. In ES you can query on myname.yourname. You can highlight it also, make facets on it....

Is it what you are asking?

David :wink:
Twitter : @dadoonet / @elasticsearchfr

Le 10 août 2012 à 09:21, Bernd Fehling bernd.fehling@googlemail.com a écrit :

Actually I don't care where to find it, I only want to have it work like copyField in Solr.
Load it into ES with one field e.g. "myname" but have it in the index in "myname" and "yourname" or if nessessary in "myname.yourname".
Exactly as it is in Solr.
From all the mailings and sparse docs I found it was stated that this can be done with "Multi Field Type".
Or not?


(Bernd Fehling) #5

Sorry, no.

But from your kind of question I assume that ES does not support the
functionality like copyField with Solr.
This is bad, especially for users trying to switch to ES.

So if I understand it right:

  • only content which is really loaded (present in the data) goes into
    _source field.
  • content which is produced during "processing" (like multi field type)
    doesn't make it into the _source field.
    It will only be indexed and therefore never show up in the result.
  • if I want it to show up in the result I have to generate it in the data
    before it gets loaded.

Right so far?


(David Pilato) #6

Yes. _source is exactly what you sent.

If you sent { "field" : "value" } you will always get back { "field" : "value" }
even if "field" is a multifield.

I don't know if you ask in your query for specific fields
(http://www.elasticsearch.org/guide/reference/api/search/fields.html
http://www.elasticsearch.org/guide/reference/api/search/fields.html ) you can
get the content of "field.multifield". I never tested it.

David.

Le 10 août 2012 à 09:56, Bernd Fehling bernd.fehling@googlemail.com a écrit :

Sorry, no.

But from your kind of question I assume that ES does not support the
functionality like copyField with Solr.
This is bad, especially for users trying to switch to ES.

So if I understand it right:

  • only content which is really loaded (present in the data) goes into _source
    field.
  • content which is produced during "processing" (like multi field type)
    doesn't make it into the _source field.
    It will only be indexed and therefore never show up in the result.
  • if I want it to show up in the result I have to generate it in the data
    before it gets loaded.

Right so far?

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


(Bernd Fehling) #7

OK, now we have clarified why I can only search on "sub-fields" of type
"multi-field" and won't get any stored data from the _source field of this
"sub-fields", because
there is no data in the _source.

And this is also the cause why ES is setting "include_in_all" always to
false for all "sub-fields" of type "multi-field" in the mapping, even if I
set it to "include_in_all" to true and load that mapping?


(David Pilato) #8

And this is also the cause why ES is setting "include_in_all" always to false
for all
"sub-fields" of type "multi-field" in the mapping, even if I set it to
"include_in_all"
to true and load that mapping?
Sorry. I don't know.

Someone?

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


(Runar Myklebust-2) #9

On Fri, Aug 10, 2012 at 11:25 AM, David Pilato david@pilato.fr wrote:

**

And this is also the cause why ES is setting "include_in_all" always to
false for all

"sub-fields" of type "multi-field" in the mapping, even if I set it to
"include_in_all"

to true and load that mapping?

Sorry. I don't know.

Someone?

If I understand your questing correctly: "Include in all" does affect the
"_all" field, not the "_source" field.
If you would like to store the value of your mutlifield, configure with
"store" : "yes". You could specify fields in queries, and when using e.g
_search?fields=* it will show all stored field. Storing of fields is
disabled by default.

greetings

Runar Myklebust
Enonic AS
Senior Developer
http://www.enonic.com/download
http://enonic.com/download


(Igor Motov) #10

Bernd,

The _all field in elasticsearch is like a special version of a Solr's
copyField that is enabled for all fields and "copies" all fields into the
_all field by default. So, your mapping would be roughly equivalent to
something like this:

So, I guess, what you are asking is why elasticserch doesn't
add ? It makes no sense to copy
the text of the field myname to _all twice. It's already there from the
myname, no need to copy it from yourname.

Did I understand your question correctly?

Igor

On Friday, August 10, 2012 9:11:24 AM UTC-4, Runar Myklebust wrote:

On Fri, Aug 10, 2012 at 11:25 AM, David Pilato <da...@pilato.fr<javascript:>

wrote:

**

And this is also the cause why ES is setting "include_in_all" always to
false for all

"sub-fields" of type "multi-field" in the mapping, even if I set it to
"include_in_all"

to true and load that mapping?

Sorry. I don't know.

Someone?

If I understand your questing correctly: "Include in all" does affect the
"_all" field, not the "_source" field.
If you would like to store the value of your mutlifield, configure with
"store" : "yes". You could specify fields in queries, and when using e.g
_search?fields=* it will show all stored field. Storing of fields is
disabled by default.

greetings

Runar Myklebust
Enonic AS
Senior Developer
http://www.enonic.com/download
http://enonic.com/download


(Clinton Gormley) #11

So, I guess, what you are asking is why elasticserch doesn't add
? It makes no sense to copy
the text of the field myname to _all twice. It's already there from
the myname, no need to copy it from yourname.

To add to this - The reason for having multi-fields is to analyze the
same data in various ways. The _all field has its own analyzer, it
doesn't copy the tokens from the other fields, just the original values
which it then analyzes itself. For this reason it doesn't make sense to
copy the values multiple times from the multi-field.

clint

Did I understand your question correctly?

Igor

On Friday, August 10, 2012 9:11:24 AM UTC-4, Runar Myklebust wrote:

    On Fri, Aug 10, 2012 at 11:25 AM, David Pilato
    <da...@pilato.fr> wrote:
            > And this is also the cause why ES is setting
            "include_in_all" always to false for all 
            
            > "sub-fields" of type "multi-field" in the mapping,
            even if I set it to "include_in_all" 
            
            > to true and load that mapping?
            
            
            Sorry. I don't know. 
            
             
            
            Someone?
            
             
            
            
    
    If I understand your questing correctly: "Include in all" does
    affect the "_all" field, not the "_source" field.
    If you would like to store the value of your mutlifield,
    configure with "store" : "yes". You could specify fields in
    queries, and when using e.g _search?fields=* it will show all
    stored field. Storing of fields is disabled by default.
    
    
    greetings
    --
    Runar Myklebust
    Enonic AS
    Senior Developer
    
    http://enonic.com/download

--


(Bernd Fehling) #12

Hi Igor,

the "store=yes" was a good move forward.
Now I have "yourname" not only indexed but also stored and I can see it
with luke in the index.
But how do I get it displayed?

I tried
curl -XGET "localhost:9200/_search?q=*&pretty=true"
but "myname.yourname" isn't in the result.

Any other ideas?

Am Freitag, 10. August 2012 16:10:49 UTC+2 schrieb Igor Motov:

Bernd,

The _all field in elasticsearch is like a special version of a Solr's
copyField that is enabled for all fields and "copies" all fields into the
_all field by default. So, your mapping would be roughly equivalent to
something like this:

So, I guess, what you are asking is why elasticserch doesn't
add ? It makes no sense to copy
the text of the field myname to _all twice. It's already there from the
myname, no need to copy it from yourname.

Did I understand your question correctly?

Igor

On Friday, August 10, 2012 9:11:24 AM UTC-4, Runar Myklebust wrote:

On Fri, Aug 10, 2012 at 11:25 AM, David Pilato da...@pilato.fr wrote:

**

And this is also the cause why ES is setting "include_in_all" always
to false for all

"sub-fields" of type "multi-field" in the mapping, even if I set it to
"include_in_all"

to true and load that mapping?

Sorry. I don't know.

Someone?

If I understand your questing correctly: "Include in all" does affect the
"_all" field, not the "_source" field.
If you would like to store the value of your mutlifield, configure with
"store" : "yes". You could specify fields in queries, and when using e.g
_search?fields=* it will show all stored field. Storing of fields is
disabled by default.

greetings

Runar Myklebust
Enonic AS
Senior Developer
http://www.enonic.com/download
http://enonic.com/download

--


(Bernd Fehling) #13

Hi Clinton,

I can accept that ES I trying to "help" me and changing some settings to
get a stable search system,
but why is it not reporting that it has changed one or some of my settings
silently in the background?
Where is the logic behind this?

Bernd

Am Samstag, 11. August 2012 12:02:23 UTC+2 schrieb Clinton Gormley:

So, I guess, what you are asking is why elasticserch doesn't add
? It makes no sense to copy
the text of the field myname to _all twice. It's already there from
the myname, no need to copy it from yourname.

To add to this - The reason for having multi-fields is to analyze the
same data in various ways. The _all field has its own analyzer, it
doesn't copy the tokens from the other fields, just the original values
which it then analyzes itself. For this reason it doesn't make sense to
copy the values multiple times from the multi-field.

clint

Did I understand your question correctly?

Igor

On Friday, August 10, 2012 9:11:24 AM UTC-4, Runar Myklebust wrote:

    On Fri, Aug 10, 2012 at 11:25 AM, David Pilato 
    <da...@pilato.fr> wrote: 
            > And this is also the cause why ES is setting 
            "include_in_all" always to false for all 
            
            > "sub-fields" of type "multi-field" in the mapping, 
            even if I set it to "include_in_all" 
            
            > to true and load that mapping? 
            
            
            Sorry. I don't know. 
            
              
            
            Someone? 
            
              
            
            
    
    If I understand your questing correctly: "Include in all" does 
    affect the "_all" field, not the "_source" field. 
    If you would like to store the value of your mutlifield, 
    configure with "store" : "yes". You could specify fields in 
    queries, and when using e.g _search?fields=* it will show all 
    stored field. Storing of fields is disabled by default. 
    
    
    greetings 
    -- 
    Runar Myklebust 
    Enonic AS 
    Senior Developer 
    
    http://enonic.com/download 

--


(Bernd Fehling) #14

Hi Runar,

thanks for your solution.
I'm just trying to understand the logic of ES which is quite different from
SOLR.

Something which is still confusing:

  • if querying with "curl -XGET "localhost:9200/_search?q=*" I get only
    _source as result.
  • if querying with "curl -XGET "localhost:9200/_search?fields=*" I get only
    stored fields as result.

But how do I manage to get both together?

Am Freitag, 10. August 2012 15:11:24 UTC+2 schrieb Runar Myklebust:

On Fri, Aug 10, 2012 at 11:25 AM, David Pilato <da...@pilato.fr<javascript:>

wrote:

**

And this is also the cause why ES is setting "include_in_all" always to
false for all

"sub-fields" of type "multi-field" in the mapping, even if I set it to
"include_in_all"

to true and load that mapping?

Sorry. I don't know.

Someone?

If I understand your questing correctly: "Include in all" does affect the
"_all" field, not the "_source" field.
If you would like to store the value of your mutlifield, configure with
"store" : "yes". You could specify fields in queries, and when using e.g
_search?fields=* it will show all stored field. Storing of fields is
disabled by default.

greetings

Runar Myklebust
Enonic AS
Senior Developer
http://www.enonic.com/download
http://enonic.com/download

--


(Runar Myklebust-2) #15

On Mon, Aug 13, 2012 at 8:31 AM, Bernd Fehling <bernd.fehling@googlemail.com

wrote:

Hi Runar,

thanks for your solution.
I'm just trying to understand the logic of ES which is quite different
from SOLR.

Something which is still confusing:

  • if querying with "curl -XGET "localhost:9200/_search?q=*" I get only
    _source as result.
  • if querying with "curl -XGET "localhost:9200/_search?fields=*" I get
    only stored fields as result.

But how do I manage to get both together?

Try:

curl -XGET "localhost:9200/_search?fields=*,_source

I guess you could also query the specific field that you are after without
storing it, just query it by name, e.g curl -XGET
"localhost:9200/_search?fields=yourname
It should then extract it on the fly, but I'm not able to test this atm.

greetings

Runar Myklebust
Enonic AS
Senior Developer

http://enonic.com/download

--


(Bernd Fehling) #16

Hi Runar,

yes it works with "fields=*,_source". Seams somehow similar like "fl=" for
SOLR.

Bernd

Am Montag, 13. August 2012 10:08:09 UTC+2 schrieb Runar Myklebust:

On Mon, Aug 13, 2012 at 8:31 AM, Bernd Fehling <bernd....@googlemail.com<javascript:>

wrote:

Hi Runar,

thanks for your solution.
I'm just trying to understand the logic of ES which is quite different
from SOLR.

Something which is still confusing:

  • if querying with "curl -XGET "localhost:9200/_search?q=*" I get only
    _source as result.
  • if querying with "curl -XGET "localhost:9200/_search?fields=*" I get
    only stored fields as result.

But how do I manage to get both together?

Try:

curl -XGET "localhost:9200/_search?fields=*,_source

I guess you could also query the specific field that you are after without
storing it, just query it by name, e.g curl -XGET
"localhost:9200/_search?fields=yourname
It should then extract it on the fly, but I'm not able to test this atm.

greetings

Runar Myklebust
Enonic AS
Senior Developer

http://enonic.com/download

--


(system) #17