Index configuration


(Yannick) #1

I'm trying to set a default index configuration in config/elastisearch.yml but for some reason it doesn't seem to get picked up. Can somebody spot the error?
The ES doesn't seem to show an error when loading. Here is the yml:

index:
analysis:
analyzer:
default_index:
tokenizer: standard
filter: [standard, lowercase, my_synonym, my_snow]
default_search:
tokenizer: standard
filter: [standard, lowercase, stop]
filter:
my_synonym:
type: synonym
synonyms_path: 'analysers/synonyms.dat'
ignore_case: true
my_snow:
type: snowball
language: English


(Shay Banon) #2

After you create the index, can you execute a get settings on it and see if it was applied to not?http://www.elasticsearch.org/guide/reference/api/admin-indices-get-settings.html.

On Saturday, May 28, 2011 at 1:21 AM, Yannick Smits wrote:

I’m trying to set a default index configuration in config/elastisearch.yml but for some reason it doesn’t seem to get picked up. Can somebody spot the error?
The ES doesn’t seem to show an error when loading. Here is the yml:

index:
analysis:
analyzer:
default_index:
tokenizer: standard
filter: [standard, lowercase, my_synonym, my_snow]
default_search:
tokenizer: standard
filter: [standard, lowercase, stop]
filter:
my_synonym:
type: synonym
synonyms_path: 'analysers/synonyms.dat'
ignore_case: true
my_snow:
type: snowball
language: English


(David Pilato) #3

Hi Yannick,

Something I found using ES.

If you have already indexed one document (with default mapping or not),
config files won't be read by ES.
So, to use mapping within config file, you have to start a clean ES cluster
with the config files put in mappings/indexName/typeName.json

Yo can also have a look at this discussion thread :
http://elasticsearch-users.115913.n3.nabble.com/id-and-mappings-tp2973448p2980327.html

Hope this helps
David

Le , Yannick Smits yannick@goyaweb.nl a écrit :

I'm trying to set a default index configuration in
config/elastisearch.yml but for some reason it doesn't seem to get picked
up. Can somebody spot the error?The ES doesn't seem to show an error when
loading. Here is the yml: index: analysis: analyzer: default_index:
tokenizer: standard filter: [standard, lowercase, my_synonym, my_snow]
default_search: tokenizer: standard filter: [standard, lowercase, stop]
filter: my_synonym: type: synonym synonyms_path: 'analysers/synonyms.dat'
ignore_case: true my_snow: type: snowball language: English


(dpilato) #4

Hi Yannick,

Something I found using ES.

If you have already indexed one document (with default mapping or not),
config files won't be read by ES.
So, to use mapping within config file, you have to start a clean ES cluster
with the config files put in mappings/indexName/typeName.json

Yo can also have a look at this discussion thread :
http://elasticsearch-users.115913.n3.nabble.com/id-and-mappings-tp2973448p2980327.html

Hope this helps
David

2011/5/28 Yannick Smits yannick@goyaweb.nl

I’m trying to set a default index configuration in config/elastisearch.yml
but for some reason it doesn’t seem to get picked up. Can somebody spot the
error?

The ES doesn’t seem to show an error when loading. Here is the yml:

index:

analysis:

analyzer:

    default_index:

        tokenizer: standard

        filter: [standard, lowercase, my_synonym, my_snow]

    default_search:

        tokenizer: standard

        filter: [standard, lowercase, stop]

filter:

    my_synonym:

        type: synonym

        synonyms_path: 'analysers/synonyms.dat'

        ignore_case: true

    my_snow:

        type: snowball

        language: English

(Yannick Smits-2) #5

This is what I get back:

{
"mms":{
"settings":{
"index.number_of_shards":"5",
"index.number_of_replicas":"1"
}
}
}

Oh and to reply to David: yes I’m aware that settings have to be applied before the index is created. So I removed the index completely using the delete command before setting the new configuration and adding documents.

From: Shay Banon [mailto:shay.banon@elasticsearch.com]
Sent: zaterdag 28 mei 2011 11:20
To: users@elasticsearch.com
Subject: Re: index configuration

After you create the index, can you execute a get settings on it and see if it was applied to not?http://www.elasticsearch.org/guide/reference/api/admin-indices-get-settings.html.

On Saturday, May 28, 2011 at 1:21 AM, Yannick Smits wrote:

I’m trying to set a default index configuration in config/elastisearch.yml but for some reason it doesn’t seem to get picked up. Can somebody spot the error?

The ES doesn’t seem to show an error when loading. Here is the yml:

index:

analysis:

analyzer:

    default_index:

        tokenizer: standard

        filter: [standard, lowercase, my_synonym, my_snow]

    default_search:

        tokenizer: standard

        filter: [standard, lowercase, stop]

filter:

    my_synonym:

        type: synonym

        synonyms_path: 'analysers/synonyms.dat'

        ignore_case: true

    my_snow:

        type: snowball

        language: English

(Shay Banon) #6

ok. how do you know that its not getting picked up? Can you gist a repro of what you are seeing?

On Saturday, May 28, 2011 at 3:34 PM, Yannick Smits wrote:

This is what I get back:

{
"mms":{
"settings":{
"index.number_of_shards":"5",
"index.number_of_replicas":"1"
}
}
}

Oh and to reply to David: yes I’m aware that settings have to be applied before the index is created. So I removed the index completely using the delete command before setting the new configuration and adding documents.

From: Shay Banon [mailto:shay.banon@elasticsearch.com]
Sent: zaterdag 28 mei 2011 11:20
To: users@elasticsearch.com (mailto:users@elasticsearch.com)
Subject: Re: index configuration

After you create the index, can you execute a get settings on it and see if it was applied to not?http://www.elasticsearch.org/guide/reference/api/admin-indices-get-settings.html.
On Saturday, May 28, 2011 at 1:21 AM, Yannick Smits wrote:

I’m trying to set a default index configuration in config/elastisearch.yml but for some reason it doesn’t seem to get picked up. Can somebody spot the error?
The ES doesn’t seem to show an error when loading. Here is the yml:

index:
analysis:
analyzer:
default_index:
tokenizer: standard
filter: [standard, lowercase, my_synonym, my_snow]
default_search:
tokenizer: standard
filter: [standard, lowercase, stop]
filter:
my_synonym:
type: synonym
synonyms_path: 'analysers/synonyms.dat'
ignore_case: true
my_snow:
type: snowball
language: English


(Yannick Smits-2) #7

I tried to search for some of the synonyms in the file which should have matched up with words in my documents, but no results. Also I tested simple search for stemming which didn’t give results either.

Is there a way to get the analysers back from ES to check if they are correctly applied using a command?

Can you explain me what gisting means?

Thanks,
Yannick

From: Shay Banon [mailto:shay.banon@elasticsearch.com]
Sent: zaterdag 28 mei 2011 15:49
To: users@elasticsearch.com
Subject: Re: index configuration

ok. how do you know that its not getting picked up? Can you gist a repro of what you are seeing?

On Saturday, May 28, 2011 at 3:34 PM, Yannick Smits wrote:

This is what I get back:

{
"mms":{
"settings":{
"index.number_of_shards":"5",
"index.number_of_replicas":"1"
}
}
}

Oh and to reply to David: yes I’m aware that settings have to be applied before the index is created. So I removed the index completely using the delete command before setting the new configuration and adding documents.

From: Shay Banon [mailto:shay.banon@elasticsearch.com]
Sent: zaterdag 28 mei 2011 11:20
To: users@elasticsearch.com
Subject: Re: index configuration

After you create the index, can you execute a get settings on it and see if it was applied to not?http://www.elasticsearch.org/guide/reference/api/admin-indices-get-settings.html.

On Saturday, May 28, 2011 at 1:21 AM, Yannick Smits wrote:

I’m trying to set a default index configuration in config/elastisearch.yml but for some reason it doesn’t seem to get picked up. Can somebody spot the error?

The ES doesn’t seem to show an error when loading. Here is the yml:

index:

analysis:

analyzer:

    default_index:

        tokenizer: standard

        filter: [standard, lowercase, my_synonym, my_snow]

    default_search:

        tokenizer: standard

        filter: [standard, lowercase, stop]

filter:

    my_synonym:

        type: synonym

        synonyms_path: 'analysers/synonyms.dat'

        ignore_case: true

    my_snow:

        type: snowball

        language: English

(Shay Banon) #8

Gist: http://gist.github.com Use the analyze API to test for the analysis.

On Saturday, May 28, 2011 at 5:12 PM, Yannick Smits wrote:

I tried to search for some of the synonyms in the file which should have matched up with words in my documents, but no results. Also I tested simple search for stemming which didn’t give results either.

Is there a way to get the analysers back from ES to check if they are correctly applied using a command?

Can you explain me what gisting means?

Thanks,
Yannick

From: Shay Banon [mailto:shay.banon@elasticsearch.com]
Sent: zaterdag 28 mei 2011 15:49
To: users@elasticsearch.com (mailto:users@elasticsearch.com)
Subject: Re: index configuration

ok. how do you know that its not getting picked up? Can you gist a repro of what you are seeing?
On Saturday, May 28, 2011 at 3:34 PM, Yannick Smits wrote:

This is what I get back:

{
"mms":{
"settings":{
"index.number_of_shards":"5",
"index.number_of_replicas":"1"
}
}
}

Oh and to reply to David: yes I’m aware that settings have to be applied before the index is created. So I removed the index completely using the delete command before setting the new configuration and adding documents.

From: Shay Banon [mailto:shay.banon@elasticsearch.com]
Sent: zaterdag 28 mei 2011 11:20
To: users@elasticsearch.com (mailto:users@elasticsearch.com)
Subject: Re: index configuration

After you create the index, can you execute a get settings on it and see if it was applied to not?http://www.elasticsearch.org/guide/reference/api/admin-indices-get-settings.html.
On Saturday, May 28, 2011 at 1:21 AM, Yannick Smits wrote:

I’m trying to set a default index configuration in config/elastisearch.yml but for some reason it doesn’t seem to get picked up. Can somebody spot the error?
The ES doesn’t seem to show an error when loading. Here is the yml:

index:
analysis:
analyzer:
default_index:
tokenizer: standard
filter: [standard, lowercase, my_synonym, my_snow]
default_search:
tokenizer: standard
filter: [standard, lowercase, stop]
filter:
my_synonym:
type: synonym
synonyms_path: 'analysers/synonyms.dat'
ignore_case: true
my_snow:
type: snowball
language: English


(Yannick Smits-2) #9

Ok, I got the analyzer settings to shop up in the index _settings request by using a JSON put. Cannot seem to set those using the YAML approach as explained below.

But even while the JSON method of setting it does seem to get through my tests show they are not working / not configured correctly.

I’ve gisted that: https://gist.github.com/1000276 https://gist.github.com/1000276

The last 2 curl requests should give 1 result but give 0 instead.

Thanks,

Yannick

From: Shay Banon [mailto:shay.banon@elasticsearch.com]
Sent: zondag 29 mei 2011 17:03
To: users@elasticsearch.com
Subject: Re: index configuration

Gist: http://gist.github.com Use the analyze API to test for the analysis.

On Saturday, May 28, 2011 at 5:12 PM, Yannick Smits wrote:

I tried to search for some of the synonyms in the file which should have matched up with words in my documents, but no results. Also I tested simple search for stemming which didn’t give results either.

Is there a way to get the analysers back from ES to check if they are correctly applied using a command?

Can you explain me what gisting means?

Thanks,
Yannick

From: Shay Banon [mailto:shay.banon@elasticsearch.com]
Sent: zaterdag 28 mei 2011 15:49
To: users@elasticsearch.com
Subject: Re: index configuration

ok. how do you know that its not getting picked up? Can you gist a repro of what you are seeing?

On Saturday, May 28, 2011 at 3:34 PM, Yannick Smits wrote:

This is what I get back:

{
"mms":{
"settings":{
"index.number_of_shards":"5",
"index.number_of_replicas":"1"
}
}
}

Oh and to reply to David: yes I’m aware that settings have to be applied before the index is created. So I removed the index completely using the delete command before setting the new configuration and adding documents.

From: Shay Banon [mailto:shay.banon@elasticsearch.com]
Sent: zaterdag 28 mei 2011 11:20
To: users@elasticsearch.com
Subject: Re: index configuration

After you create the index, can you execute a get settings on it and see if it was applied to not?http://www.elasticsearch.org/guide/reference/api/admin-indices-get-settings.html.

On Saturday, May 28, 2011 at 1:21 AM, Yannick Smits wrote:

I’m trying to set a default index configuration in config/elastisearch.yml but for some reason it doesn’t seem to get picked up. Can somebody spot the error?

The ES doesn’t seem to show an error when loading. Here is the yml:

index:

analysis:

analyzer:

    default_index:

        tokenizer: standard

        filter: [standard, lowercase, my_synonym, my_snow]

    default_search:

        tokenizer: standard

        filter: [standard, lowercase, stop]

filter:

    my_synonym:

        type: synonym

        synonyms_path: 'analysers/synonyms.dat'

        ignore_case: true

    my_snow:

        type: snowball

        language: English

(Yannick Smits-2) #10

Could anybody try my gist? This issue is still unresolved.

From: Yannick Smits [mailto:mailinglists@goyaweb.nl]
Sent: dinsdag 31 mei 2011 12:31
To: users@elasticsearch.com
Subject: RE: index configuration

Ok, I got the analyzer settings to shop up in the index _settings request by using a JSON put. Cannot seem to set those using the YAML approach as explained below.

But even while the JSON method of setting it does seem to get through my tests show they are not working / not configured correctly.

I’ve gisted that: https://gist.github.com/1006503 https://gist.github.com/1006503

The last 2 curl requests should give 1 result but give 0 instead.

Thanks,

Yannick

From: Shay Banon [mailto:shay.banon@elasticsearch.com]
Sent: zondag 29 mei 2011 17:03
To: users@elasticsearch.com
Subject: Re: index configuration

Gist: http://gist.github.com Use the analyze API to test for the analysis.

On Saturday, May 28, 2011 at 5:12 PM, Yannick Smits wrote:

I tried to search for some of the synonyms in the file which should have matched up with words in my documents, but no results. Also I tested simple search for stemming which didn’t give results either.

Is there a way to get the analysers back from ES to check if they are correctly applied using a command?

Can you explain me what gisting means?

Thanks,
Yannick

From: Shay Banon [mailto:shay.banon@elasticsearch.com]
Sent: zaterdag 28 mei 2011 15:49
To: users@elasticsearch.com
Subject: Re: index configuration

ok. how do you know that its not getting picked up? Can you gist a repro of what you are seeing?

On Saturday, May 28, 2011 at 3:34 PM, Yannick Smits wrote:

This is what I get back:

{
"mms":{
"settings":{
"index.number_of_shards":"5",
"index.number_of_replicas":"1"
}
}
}

Oh and to reply to David: yes I’m aware that settings have to be applied before the index is created. So I removed the index completely using the delete command before setting the new configuration and adding documents.

From: Shay Banon [mailto:shay.banon@elasticsearch.com]
Sent: zaterdag 28 mei 2011 11:20
To: users@elasticsearch.com
Subject: Re: index configuration

After you create the index, can you execute a get settings on it and see if it was applied to not?http://www.elasticsearch.org/guide/reference/api/admin-indices-get-settings.html.

On Saturday, May 28, 2011 at 1:21 AM, Yannick Smits wrote:

I’m trying to set a default index configuration in config/elastisearch.yml but for some reason it doesn’t seem to get picked up. Can somebody spot the error?

The ES doesn’t seem to show an error when loading. Here is the yml:

index:

analysis:

analyzer:

    default_index:

        tokenizer: standard

        filter: [standard, lowercase, my_synonym, my_snow]

    default_search:

        tokenizer: standard

        filter: [standard, lowercase, stop]

filter:

    my_synonym:

        type: synonym

        synonyms_path: 'analysers/synonyms.dat'

        ignore_case: true

    my_snow:

        type: snowball

        language: English

(Clinton Gormley) #11

Hi Yannick

On Fri, 2011-06-03 at 17:18 +0200, Yannick Smits wrote:

Could anybody try my gist? This issue is still unresolved.

https://gist.github.com/1006503

You're using the snowball filter on text that you are indexing but then:

  1. you're using a term query, which doesn't get analysed, instead of
    a text|field|query_string query

  2. even if you were using a text|field|query_string query, it probably
    would fail as well, because your default search analyzer doesn't
    include the snowball filter.

You almost always want to use the same analyzer for indexing and for
searching

clint


(Yannick Smits-2) #12

Hi Clint,

Thanks for pointing this out to me. Now using a text query and making the index and search analyzers equal everything works as expected. The only thing I can't seem to get to work is setting the default analyzers inside the elasticsearch.yml. But using the json put method it works well enough for me now.

Thanks!

-----Original Message-----
From: Clinton Gormley [mailto:clinton@iannounce.co.uk]
Sent: vrijdag 3 juni 2011 17:33
To: users@elasticsearch.com
Subject: RE: index configuration

Hi Yannick

On Fri, 2011-06-03 at 17:18 +0200, Yannick Smits wrote:

Could anybody try my gist? This issue is still unresolved.

https://gist.github.com/1006503

You're using the snowball filter on text that you are indexing but then:

  1. you're using a term query, which doesn't get analysed, instead of
    a text|field|query_string query

  2. even if you were using a text|field|query_string query, it probably
    would fail as well, because your default search analyzer doesn't
    include the snowball filter.

You almost always want to use the same analyzer for indexing and for searching

clint


(Yannick Smits-2) #13

You almost always want to use the same analyzer for indexing and for searching

Could you elaborate on why that is? I would imagine if the synonyms are already applied to the index that it doesn't make sense to apply the synonyms on the search query, because they are already matched in the index.

-----Original Message-----
From: Yannick Smits [mailto:mailinglists@goyaweb.nl]
Sent: zaterdag 4 juni 2011 16:26
To: users@elasticsearch.com
Subject: RE: index configuration

Hi Clint,

Thanks for pointing this out to me. Now using a text query and making the index and search analyzers equal everything works as expected. The only thing I can't seem to get to work is setting the default analyzers inside the elasticsearch.yml. But using the json put method it works well enough for me now.

Thanks!

-----Original Message-----
From: Clinton Gormley [mailto:clinton@iannounce.co.uk]
Sent: vrijdag 3 juni 2011 17:33
To: users@elasticsearch.com
Subject: RE: index configuration

Hi Yannick

On Fri, 2011-06-03 at 17:18 +0200, Yannick Smits wrote:

Could anybody try my gist? This issue is still unresolved.

https://gist.github.com/1006503

You're using the snowball filter on text that you are indexing but then:

  1. you're using a term query, which doesn't get analysed, instead of
    a text|field|query_string query

  2. even if you were using a text|field|query_string query, it probably
    would fail as well, because your default search analyzer doesn't
    include the snowball filter.

You almost always want to use the same analyzer for indexing and for searching

clint


(James Cook) #14

How do you specify which analyzer to use for searching?

On Mon, Jun 6, 2011 at 10:23 AM, Yannick Smits mailinglists@goyaweb.nlwrote:

You almost always want to use the same analyzer for indexing and for
searching

Could you elaborate on why that is? I would imagine if the synonyms are
already applied to the index that it doesn't make sense to apply the
synonyms on the search query, because they are already matched in the index.

-----Original Message-----
From: Yannick Smits [mailto:mailinglists@goyaweb.nl]
Sent: zaterdag 4 juni 2011 16:26
To: users@elasticsearch.com
Subject: RE: index configuration

Hi Clint,

Thanks for pointing this out to me. Now using a text query and making the
index and search analyzers equal everything works as expected. The only
thing I can't seem to get to work is setting the default analyzers inside
the elasticsearch.yml. But using the json put method it works well enough
for me now.

Thanks!

-----Original Message-----
From: Clinton Gormley [mailto:clinton@iannounce.co.uk]
Sent: vrijdag 3 juni 2011 17:33
To: users@elasticsearch.com
Subject: RE: index configuration

Hi Yannick

On Fri, 2011-06-03 at 17:18 +0200, Yannick Smits wrote:

Could anybody try my gist? This issue is still unresolved.

https://gist.github.com/1006503

You're using the snowball filter on text that you are indexing but then:

  1. you're using a term query, which doesn't get analysed, instead of
    a text|field|query_string query

  2. even if you were using a text|field|query_string query, it probably
    would fail as well, because your default search analyzer doesn't
    include the snowball filter.

You almost always want to use the same analyzer for indexing and for
searching

clint


(Yannick Smits-2) #15

From: James Cook [mailto:jcook@tracermedia.com]
Sent: maandag 6 juni 2011 16:48
To: users@elasticsearch.com
Subject: Re: index configuration

How do you specify which analyzer to use for searching?

On Mon, Jun 6, 2011 at 10:23 AM, Yannick Smits mailinglists@goyaweb.nl
wrote:

You almost always want to use the same analyzer for indexing and for
searching

Could you elaborate on why that is? I would imagine if the synonyms are
already applied to the index that it doesn't make sense to apply the
synonyms on the search query, because they are already matched in the index.

-----Original Message-----
From: Yannick Smits [mailto:mailinglists@goyaweb.nl]

Sent: zaterdag 4 juni 2011 16:26
To: users@elasticsearch.com
Subject: RE: index configuration

Hi Clint,

Thanks for pointing this out to me. Now using a text query and making the
index and search analyzers equal everything works as expected. The only
thing I can't seem to get to work is setting the default analyzers inside
the elasticsearch.yml. But using the json put method it works well enough
for me now.

Thanks!

-----Original Message-----
From: Clinton Gormley [mailto:clinton@iannounce.co.uk]
Sent: vrijdag 3 juni 2011 17:33
To: users@elasticsearch.com
Subject: RE: index configuration

Hi Yannick

On Fri, 2011-06-03 at 17:18 +0200, Yannick Smits wrote:

Could anybody try my gist? This issue is still unresolved.

https://gist.github.com/1006503

You're using the snowball filter on text that you are indexing but then:

  1. you're using a term query, which doesn't get analysed, instead of
    a text|field|query_string query

  2. even if you were using a text|field|query_string query, it probably
    would fail as well, because your default search analyzer doesn't
    include the snowball filter.

You almost always want to use the same analyzer for indexing and for
searching

clint


(Greg Brown) #16

Could you elaborate on why that is? I would imagine if the synonyms are already applied to the index that it doesn't make sense to apply the synonyms on the search query, because they are already matched in the index.

You're right that for synonyms this probably doesn't matter; however,
for the snowball stemmer you are using you definitely should apply it
to both the query and the index since the the stemmer will cause only
the stemmed version of words to be indexed, while a user will often
search with words that are not stemmed. So for instance, a search for
the word "indexing" will not match anything if all documents in the
index have had the word "indexing" stemmed to "index".

Generally good practice to just use the same analyzer for both the
query and indexing to avoid these types of problems.

On Jun 6, 8:23 am, Yannick Smits mailingli...@goyaweb.nl wrote:

You almost always want to use the same analyzer for indexing and for searching

Could you elaborate on why that is? I would imagine if the synonyms are already applied to the index that it doesn't make sense to apply the synonyms on the search query, because they are already matched in the index.

-----Original Message-----
From: Yannick Smits [mailto:mailingli...@goyaweb.nl]
Sent: zaterdag 4 juni 2011 16:26
To: us...@elasticsearch.com
Subject: RE: index configuration

Hi Clint,

Thanks for pointing this out to me. Now using a text query and making the index and search analyzers equal everything works as expected. The only thing I can't seem to get to work is setting the default analyzers inside the elasticsearch.yml. But using the json put method it works well enough for me now.

Thanks!

-----Original Message-----
From: Clinton Gormley [mailto:clin...@iannounce.co.uk]
Sent: vrijdag 3 juni 2011 17:33
To: us...@elasticsearch.com
Subject: RE: index configuration

Hi Yannick

On Fri, 2011-06-03 at 17:18 +0200, Yannick Smits wrote:

Could anybody try my gist? This issue is still unresolved.

https://gist.github.com/1006503

You're using the snowball filter on text that you are indexing but then:

  1. you're using a term query, which doesn't get analysed, instead of
    a text|field|query_string query

  2. even if you were using a text|field|query_string query, it probably
    would fail as well, because your default search analyzer doesn't
    include the snowball filter.

You almost always want to use the same analyzer for indexing and for searching

clint


(system) #17