Asciifolding analyzer

Carlos_Eduardo_da_Fo · July 27, 2011, 10:22pm

Hi, I've done a lot of research on groups and ES guides on how to use this
analyzer but somehow it is not working.

Here's my elasticsearch.yml:

index:
analysis:
analyzer:
standard:
tokenizer: standard
type: standard
filter: [standard, lowercase, stop, asciifolding]

I based this link
http://www.elasticsearch.org/guide/reference/index-modules/analysis/ -
aliasing section

I already stopped ES, removed the data/ dir and indexed everything after
changing the file. Still not working.

My ES version is 1.17.0

What may be wrong?

Thanks,

--
Carlos Eduardo da Fonseca

kimchy · July 28, 2011, 12:53pm

You change the standard analyzer, this means that in the mapping, if you set
for a field explicitly to use the standard analyzer (set
analyzer="standard") then it will use it.

If you want to change the default analyzer, then replace the name standard
with default.

On Thu, Jul 28, 2011 at 1:22 AM, Carlos Eduardo da Fonseca <
eduardongu@gmail.com> wrote:

Hi, I've done a lot of research on groups and ES guides on how to use this
analyzer but somehow it is not working.

Here's my elasticsearch.yml:

index:
analysis:
analyzer:
standard:
tokenizer: standard
type: standard
filter: [standard, lowercase, stop, asciifolding]

I based this link
Elasticsearch Platform — Find real-time answers at scale | Elastic -
aliasing section

I already stopped ES, removed the data/ dir and indexed everything after
changing the file. Still not working.

My ES version is 1.17.0

What may be wrong?

Thanks,

--
Carlos Eduardo da Fonseca

Carlos_Eduardo_da_Fo · July 28, 2011, 1:09pm

Hi Shay, thanks for the answer.

I changed it to "default", restarted ES (cleaning data/) and reindexed, but
it doesn't worked...

elasticsearch.yml:

index:
analysis:
analyzer:
default:
tokenizer: standard
type: standard
filter: [standard, lowercase, stop, asciifolding]

On Thu, Jul 28, 2011 at 9:53 AM, Shay Banon kimchy@gmail.com wrote:

You change the standard analyzer, this means that in the mapping, if you
set for a field explicitly to use the standard analyzer (set
analyzer="standard") then it will use it.

If you want to change the default analyzer, then replace the name
standard with default.

On Thu, Jul 28, 2011 at 1:22 AM, Carlos Eduardo da Fonseca <
eduardongu@gmail.com> wrote:

Hi, I've done a lot of research on groups and ES guides on how to use this
analyzer but somehow it is not working.

Here's my elasticsearch.yml:

index:
analysis:
analyzer:
standard:
tokenizer: standard
type: standard
filter: [standard, lowercase, stop, asciifolding]

I based this link
Elasticsearch Platform — Find real-time answers at scale | Elastic -
aliasing section

I already stopped ES, removed the data/ dir and indexed everything after
changing the file. Still not working.

My ES version is 1.17.0

What may be wrong?

Thanks,

--
Carlos Eduardo da Fonseca

--
Carlos Eduardo da Fonseca

kimchy · July 28, 2011, 1:36pm

Gist an example then of why it does not work. One way to make sure it has
taked affect on indices created is to issue get settings and see if its
there.

On Thu, Jul 28, 2011 at 4:09 PM, Carlos Eduardo da Fonseca <
eduardongu@gmail.com> wrote:

Hi Shay, thanks for the answer.

I changed it to "default", restarted ES (cleaning data/) and reindexed, but
it doesn't worked...

elasticsearch.yml:

index:
analysis:
analyzer:
default:
tokenizer: standard
type: standard
filter: [standard, lowercase, stop, asciifolding]

On Thu, Jul 28, 2011 at 9:53 AM, Shay Banon kimchy@gmail.com wrote:

You change the standard analyzer, this means that in the mapping, if you
set for a field explicitly to use the standard analyzer (set
analyzer="standard") then it will use it.

If you want to change the default analyzer, then replace the name
standard with default.

On Thu, Jul 28, 2011 at 1:22 AM, Carlos Eduardo da Fonseca <
eduardongu@gmail.com> wrote:

Hi, I've done a lot of research on groups and ES guides on how to use
this analyzer but somehow it is not working.

Here's my elasticsearch.yml:

index:
analysis:
analyzer:
standard:
tokenizer: standard
type: standard
filter: [standard, lowercase, stop, asciifolding]

I based this link
Elasticsearch Platform — Find real-time answers at scale | Elastic -
aliasing section

I already stopped ES, removed the data/ dir and indexed everything after
changing the file. Still not working.

My ES version is 1.17.0

What may be wrong?

Thanks,

--
Carlos Eduardo da Fonseca

--
Carlos Eduardo da Fonseca

Carlos_Eduardo_da_Fo · July 28, 2011, 2:33pm

Ok, here's a gist with index settings and everything I did after creating
the index with settings via PUT:

gist.github.com

https://gist.github.com/cedufca/1a92fe00ddbd71806892

gistfile1.txt

$ curl -XGET 'http://localhost:9200/generics/_settings'
> {"generics":{"settings":{"index.number_of_shards":"5","index.number_of_replicas":"1"}}}

Then I removed data/ and did:
$ curl -XPUT 'http://localhost:9200/generics' -d '
index:
  analysis:
    analyzer:
      default:
        tokenizer: standard

This file has been truncated. show original

Vladimir_Shkurin · July 28, 2011, 7:41pm

I'm using this settings and asciifolding works:

gist.github.com

https://gist.github.com/vshkurin/1109136

gistfile1.bat

#  Index
---------------------------------------------------------------------

curl -XPUT http://localhost:9200/pictures/ -d '
{
    "settings": {
        "analysis": {
            "analyzer": {
                "index_analyzer": {
                    "tokenizer": "standard",

This file has been truncated. show original

/_search?q=title:botao* - OK
/_search?q=botao* - OK

You can remove the "my_delimiter", "porter_stem" and use these
settings, just write in your mappings for field-type "string" for
"index_analyzer" and "search_analyzer" your custom analyzer.

Carlos_Eduardo_da_Fo · July 28, 2011, 8:57pm

That worked! thanks a lot!!!

On Thu, Jul 28, 2011 at 4:41 PM, Vladimir Shkurin vshkurin@gmail.comwrote:

I'm using this settings and asciifolding works:

ElasticSearch Test Queries · GitHub

/_search?q=title:botao* - OK
/_search?q=botao* - OK

You can remove the "my_delimiter", "porter_stem" and use these
settings, just write in your mappings for field-type "string" for
"index_analyzer" and "search_analyzer" your custom analyzer.

--
Carlos Eduardo da Fonseca

kimchy · July 29, 2011, 6:37am

Strange that it worked :). By default, wildcard and prefix using the
query_string query (which is what q= is translated to) don't get analyzed.
Elasticsearch Platform — Find real-time answers at scale | Elastic.
You can pass analyze_wildcard set to true, which will cause wildcards to be
analyzed as well (though its tricky).

Having said that, the analyze_wildcard parameter is not exposed when using
?q=..., though it can be added easily, something like:
?q=analyze_wildcard=true&q=... .

Most times, a better option is to use the text family of queries:
Elasticsearch Platform — Find real-time answers at scale | Elastic.

On Thu, Jul 28, 2011 at 11:57 PM, Carlos Eduardo da Fonseca <
eduardongu@gmail.com> wrote:

That worked! thanks a lot!!!

On Thu, Jul 28, 2011 at 4:41 PM, Vladimir Shkurin vshkurin@gmail.comwrote:

I'm using this settings and asciifolding works:

ElasticSearch Test Queries · GitHub

/_search?q=title:botao* - OK
/_search?q=botao* - OK

You can remove the "my_delimiter", "porter_stem" and use these
settings, just write in your mappings for field-type "string" for
"index_analyzer" and "search_analyzer" your custom analyzer.

--
Carlos Eduardo da Fonseca

Carlos_Eduardo_da_Fo · July 29, 2011, 11:41am

Shay, I searched using text and field queries, forgot to mention. I think it
wasn't working before because of missing mappings...

On Fri, Jul 29, 2011 at 3:37 AM, Shay Banon kimchy@gmail.com wrote:

Strange that it worked :). By default, wildcard and prefix using the
query_string query (which is what q= is translated to) don't get analyzed.
Elasticsearch Platform — Find real-time answers at scale | Elastic.
You can pass analyze_wildcard set to true, which will cause wildcards to be
analyzed as well (though its tricky).

Having said that, the analyze_wildcard parameter is not exposed when using
?q=..., though it can be added easily, something like:
?q=analyze_wildcard=true&q=... .

Most times, a better option is to use the text family of queries:
Elasticsearch Platform — Find real-time answers at scale | Elastic.

On Thu, Jul 28, 2011 at 11:57 PM, Carlos Eduardo da Fonseca <
eduardongu@gmail.com> wrote:

That worked! thanks a lot!!!

On Thu, Jul 28, 2011 at 4:41 PM, Vladimir Shkurin vshkurin@gmail.comwrote:

I'm using this settings and asciifolding works:

ElasticSearch Test Queries · GitHub

/_search?q=title:botao* - OK
/_search?q=botao* - OK

You can remove the "my_delimiter", "porter_stem" and use these
settings, just write in your mappings for field-type "string" for
"index_analyzer" and "search_analyzer" your custom analyzer.

--
Carlos Eduardo da Fonseca

--
Carlos Eduardo da Fonseca