Elision Token Filter with custom analyzer?

Sam_4 · February 3, 2012, 6:22am

Hi

I have applied Elision Token Filter while indexing and searching,i
created an index for “l’avion” and searched for "avion" and got the
hits as “l’avion”.
this is correct and i believe that analyzer are done correctly for
Elision.
but when i search for “l’avion” i got zero hits.i was expecting one
hits.because i am searching the same which i indexed.

why i couldn't get the hit? if i have misunderstood the concept then
make it correct.

used custom analyzer settings :

{
"index" : {
"analysis" : {
"analyzer" : {
"manualindexanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
},
"manualsearchanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
}
},
"filter" : {
"myelision":{
"type" : "elision",
"articles" :["l", "m", "t", "qu", "n", "s", "j"]
}
}
}
}
}

Mapping :

{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

to put mapping i have used following java code:

PutMappingResponse actionGet =
client.admin().indices().preparePutMapping(indexName).setType(indexType).setSource(analyzerMapping).execute().actionGet();
boolean acknowledged = actionGet.getAcknowledged();
logger.debug("put mapping acknowledged::" + acknowledged);

Thanks

kimchy · February 5, 2012, 11:30am

If the search analyzer is configured with the same elision filter, then it will be applied when searching as well, but only on queries that do analysis on the search text (query_string, text, field). Which query did you execute?

On Friday, February 3, 2012 at 8:22 AM, sam wrote:

Hi

I have applied Elision Token Filter while indexing and searching,i
created an index for “l’avion” and searched for "avion" and got the
hits as “l’avion”.
this is correct and i believe that analyzer are done correctly for
Elision.
but when i search for “l’avion” i got zero hits.i was expecting one
hits.because i am searching the same which i indexed.

why i couldn't get the hit? if i have misunderstood the concept then
make it correct.

used custom analyzer settings :

{
"index" : {
"analysis" : {
"analyzer" : {
"manualindexanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
},
"manualsearchanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
}
},
"filter" : {
"myelision":{
"type" : "elision",
"articles" :["l", "m", "t", "qu", "n", "s", "j"]
}
}
}
}
}

Mapping :

{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

to put mapping i have used following java code:

PutMappingResponse actionGet =
client.admin().indices().preparePutMapping(indexName).setType(indexType).setSource(analyzerMapping).execute().actionGet();
boolean acknowledged = actionGet.getAcknowledged();
logger.debug("put mapping acknowledged::" + acknowledged);

Thanks

Sam_4 · February 7, 2012, 4:54am

Thanks for reply.

Mapping :
{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

Here I have putting mapping to configured the index_analyzer and
search_analyzer,so it is not necessary to explicit define analyzer
when doing a search.
In my case which i pointed out in above question i have used
index_analyzer and search_analyzer both are same when index is
created.for searching i have used following search query:

         QueryStringQueryBuilder querystring = new

QueryStringQueryBuilder(searchData);
SearchResponse searchResponse =
client.prepareSearch(indexName).setTypes(indexType).setSearchType(SearchType.DEFAULT)
.setQuery(querystring.field("Description")).setExplain(true).setSize(100).execute().actionGet();

Thanks

On Feb 5, 4:30 pm, Shay Banon kim...@gmail.com wrote:

If the search analyzer is configured with the same elision filter, then it will be applied when searching as well, but only on queries that do analysis on the search text (query_string, text, field). Which query did you execute?

On Friday, February 3, 2012 at 8:22 AM, sam wrote:

Hi

I have applied Elision Token Filter while indexing and searching,i
created an index for “l’avion” and searched for "avion" and got the
hits as “l’avion”.
this is correct and i believe that analyzer are done correctly for
Elision.
but when i search for “l’avion” i got zero hits.i was expecting one
hits.because i am searching the same which i indexed.

why i couldn't get the hit? if i have misunderstood the concept then
make it correct.

used custom analyzer settings :

{
"index" : {
"analysis" : {
"analyzer" : {
"manualindexanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
},
"manualsearchanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
}
},
"filter" : {
"myelision":{
"type" : "elision",
"articles" :["l", "m", "t", "qu", "n", "s", "j"]
}
}
}
}
}

Mapping :

{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

to put mapping i have used following java code:

PutMappingResponse actionGet =
client.admin().indices().preparePutMapping(indexName).setType(indexType).se tSource(analyzerMapping).execute().actionGet();
boolean acknowledged = actionGet.getAcknowledged();
logger.debug("put mapping acknowledged::" + acknowledged);

Thanks

kimchy · February 7, 2012, 11:45am

Here is a sample of where its working: gist:1759303 · GitHub.

On Tuesday, February 7, 2012 at 6:54 AM, sam wrote:

Thanks for reply.

Mapping :
{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

Here I have putting mapping to configured the index_analyzer and
search_analyzer,so it is not necessary to explicit define analyzer
when doing a search.
In my case which i pointed out in above question i have used
index_analyzer and search_analyzer both are same when index is
created.for searching i have used following search query:

QueryStringQueryBuilder querystring = new
QueryStringQueryBuilder(searchData);
SearchResponse searchResponse =
client.prepareSearch(indexName).setTypes(indexType).setSearchType(SearchType.DEFAULT)
.setQuery(querystring.field("Description")).setExplain(true).setSize(100).execute().actionGet();

Thanks

On Feb 5, 4:30 pm, Shay Banon <kim...@gmail.com (http://gmail.com)> wrote:

If the search analyzer is configured with the same elision filter, then it will be applied when searching as well, but only on queries that do analysis on the search text (query_string, text, field). Which query did you execute?

On Friday, February 3, 2012 at 8:22 AM, sam wrote:

Hi

I have applied Elision Token Filter while indexing and searching,i
created an index for “l’avion” and searched for "avion" and got the
hits as “l’avion”.
this is correct and i believe that analyzer are done correctly for
Elision.
but when i search for “l’avion” i got zero hits.i was expecting one
hits.because i am searching the same which i indexed.

why i couldn't get the hit? if i have misunderstood the concept then
make it correct.

used custom analyzer settings :

{
"index" : {
"analysis" : {
"analyzer" : {
"manualindexanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
},
"manualsearchanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
}
},
"filter" : {
"myelision":{
"type" : "elision",
"articles" :["l", "m", "t", "qu", "n", "s", "j"]
}
}
}
}
}

Mapping :

{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

to put mapping i have used following java code:

PutMappingResponse actionGet =
client.admin().indices().preparePutMapping(indexName).setType(indexType).se tSource(analyzerMapping).execute().actionGet();
boolean acknowledged = actionGet.getAcknowledged();
logger.debug("put mapping acknowledged::" + acknowledged);

Thanks

Sam_4 · February 9, 2012, 5:43am

Thanks for reply.yes it's working. I figured out the problem, i added
logical analyzer name for searching, below is the used code:

QueryStringQueryBuilder querystring = new
QueryStringQueryBuilder(searchData).analyzer("manualsearchanalyzer");
SearchResponse searchResponse =
client.prepareSearch(indexName).setTypes(indexType).setSearchType(SearchTyp
e.DEFAULT)
.setQuery(querystring.field("Description")).setExplain(true).setSize(100).e
xecute().actionGet();

and previoulsy i was not giving analyzer name for searching.

I have read following document Elasticsearch Platform — Find real-time answers at scale | Elastic
that say that defining the mapping there is no need to define
explicit mapping for both cases index and search, and I have applied
this scheme. but my case it's working in indexing but not working on
searching. I needed to explicitly define search analyzer when i
searched for the given problem in this post i.e. I'avion .

What could be the reason that mapping is not applied for searching?

Thanks

On Feb 7, 4:45 pm, Shay Banon kim...@gmail.com wrote:

Here is a sample of where its working:gist:1759303 · GitHub.

On Tuesday, February 7, 2012 at 6:54 AM, sam wrote:

Thanks for reply.

Mapping :
{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

Here I have putting mapping to configured the index_analyzer and
search_analyzer,so it is not necessary to explicit define analyzer
when doing a search.
In my case which i pointed out in above question i have used
index_analyzer and search_analyzer both are same when index is
created.for searching i have used following search query:

QueryStringQueryBuilder querystring = new
QueryStringQueryBuilder(searchData);
SearchResponse searchResponse =
client.prepareSearch(indexName).setTypes(indexType).setSearchType(SearchTyp e.DEFAULT)
.setQuery(querystring.field("Description")).setExplain(true).setSize(100).e xecute().actionGet();

Thanks

On Feb 5, 4:30 pm, Shay Banon <kim...@gmail.com (http://gmail.com)> wrote:

If the search analyzer is configured with the same elision filter, then it will be applied when searching as well, but only on queries that do analysis on the search text (query_string, text, field). Which query did you execute?

On Friday, February 3, 2012 at 8:22 AM, sam wrote:

Hi

I have applied Elision Token Filter while indexing and searching,i
created an index for “l’avion” and searched for "avion" and got the
hits as “l’avion”.
this is correct and i believe that analyzer are done correctly for
Elision.
but when i search for “l’avion” i got zero hits.i was expecting one
hits.because i am searching the same which i indexed.

why i couldn't get the hit? if i have misunderstood the concept then
make it correct.

used custom analyzer settings :

{
"index" : {
"analysis" : {
"analyzer" : {
"manualindexanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
},
"manualsearchanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
}
},
"filter" : {
"myelision":{
"type" : "elision",
"articles" :["l", "m", "t", "qu", "n", "s", "j"]
}
}
}
}
}

Mapping :

{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

to put mapping i have used following java code:

PutMappingResponse actionGet =
client.admin().indices().preparePutMapping(indexName).setType(indexType).se tSource(analyzerMapping).execute().actionGet();
boolean acknowledged = actionGet.getAcknowledged();
logger.debug("put mapping acknowledged::" + acknowledged);

Thanks

kimchy · February 9, 2012, 8:29am

So now the code you posted is not the one you really used? This is annoying. If you want, I can answer the question you posted, but create a gist similar to what I created that exhibits what you expect, and I can have a look at it.

On Thursday, February 9, 2012 at 7:43 AM, sam wrote:

Thanks for reply.yes it's working. I figured out the problem, i added
logical analyzer name for searching, below is the used code:

QueryStringQueryBuilder querystring = new
QueryStringQueryBuilder(searchData).analyzer("manualsearchanalyzer");
SearchResponse searchResponse =
client.prepareSearch(indexName).setTypes(indexType).setSearchType(SearchTyp
e.DEFAULT)
.setQuery(querystring.field("Description")).setExplain(true).setSize(100).e
xecute().actionGet();

and previoulsy i was not giving analyzer name for searching.

I have read following document Elasticsearch Platform — Find real-time answers at scale | Elastic
that say that defining the mapping there is no need to define
explicit mapping for both cases index and search, and I have applied
this scheme. but my case it's working in indexing but not working on
searching. I needed to explicitly define search analyzer when i
searched for the given problem in this post i.e. I'avion .

What could be the reason that mapping is not applied for searching?

Thanks

On Feb 7, 4:45 pm, Shay Banon <kim...@gmail.com (http://gmail.com)> wrote:

Here is a sample of where its working:gist:1759303 · GitHub.

On Tuesday, February 7, 2012 at 6:54 AM, sam wrote:

Thanks for reply.

Mapping :
{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

Here I have putting mapping to configured the index_analyzer and
search_analyzer,so it is not necessary to explicit define analyzer
when doing a search.
In my case which i pointed out in above question i have used
index_analyzer and search_analyzer both are same when index is
created.for searching i have used following search query:

QueryStringQueryBuilder querystring = new
QueryStringQueryBuilder(searchData);
SearchResponse searchResponse =
client.prepareSearch(indexName).setTypes(indexType).setSearchType(SearchTyp e.DEFAULT)
.setQuery(querystring.field("Description")).setExplain(true).setSize(100).e xecute().actionGet();

Thanks

On Feb 5, 4:30 pm, Shay Banon <kim...@gmail.com (http://gmail.com)> wrote:

If the search analyzer is configured with the same elision filter, then it will be applied when searching as well, but only on queries that do analysis on the search text (query_string, text, field). Which query did you execute?

On Friday, February 3, 2012 at 8:22 AM, sam wrote:

Hi

I have applied Elision Token Filter while indexing and searching,i
created an index for “l’avion” and searched for "avion" and got the
hits as “l’avion”.
this is correct and i believe that analyzer are done correctly for
Elision.
but when i search for “l’avion” i got zero hits.i was expecting one
hits.because i am searching the same which i indexed.

why i couldn't get the hit? if i have misunderstood the concept then
make it correct.

used custom analyzer settings :

{
"index" : {
"analysis" : {
"analyzer" : {
"manualindexanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
},
"manualsearchanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
}
},
"filter" : {
"myelision":{
"type" : "elision",
"articles" :["l", "m", "t", "qu", "n", "s", "j"]
}
}
}
}
}

Mapping :

{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

to put mapping i have used following java code:

PutMappingResponse actionGet =
client.admin().indices().preparePutMapping(indexName).setType(indexType).se tSource(analyzerMapping).execute().actionGet();
boolean acknowledged = actionGet.getAcknowledged();
logger.debug("put mapping acknowledged::" + acknowledged);

Thanks

Sam_4 · February 16, 2012, 8:01am

sorry for inconvenient.i figured out the problem, i added logical
analyzer name for searching,i have following gist :

gist.github.com

https://gist.github.com/sameek/1843065

customanalyzer.json

{ 
  "index" : { 
        "analysis" : { 
                        "analyzer" : { 
                                "manualindexanalyzer" : { 
                                                        "type":"custom", 
                                                        "tokenizer" : "whitespace", 
                                                        "filter" : ["lowercase","asciifolding","length","myelision"], 
                                                        "char_filter" :["html_strip"] 
                                                                  },

This file has been truncated. show original

mapping.json

Mapping : 
{ 
"test_type" : { 
  "index_analyzer" : "manualindexanalyzer", 
"search_analyzer" : "manualsearchanalyzer" 
              } 
}

search.java

//used code for putting mapping

String indexName="idx_test";
String inddxType="test_type";

PutMappingResponse actionGet = 
client.admin().indices().preparePutMapping(indexName).setType(indexType).se tSource(analyzerMapping).execute().actionGet(); 
            boolean acknowledged = actionGet.getAcknowledged(); 
            logger.debug("put mapping acknowledged::" + acknowledged);

This file has been truncated. show original

and previoulsy i was not giving analyzer name for searching.
I have read following document Elasticsearch Platform — Find real-time answers at scale | Elastic...
that say that defining the mapping there is no need to define
explicit mapping for both cases index and search, and I have applied
this scheme. but my case it's working in indexing but not working on
searching. I needed to explicitly define search analyzer when i
searched for the given problem in this post i.e. I'avion .
What could be the reason that mapping is not applied for searching?

Thanks

On Feb 9, 1:29 pm, Shay Banon kim...@gmail.com wrote:

So now the code you posted is not the one you really used? This is annoying. If you want, I can answer the question you posted, but create a gist similar to what I created that exhibits what you expect, and I can have a look at it.

On Thursday, February 9, 2012 at 7:43 AM, sam wrote:

Thanks for reply.yes it's working. I figured out the problem, i added
logical analyzer name for searching, below is the used code:

QueryStringQueryBuilder querystring = new
QueryStringQueryBuilder(searchData).analyzer("manualsearchanalyzer");
SearchResponse searchResponse =
client.prepareSearch(indexName).setTypes(indexType).setSearchType(SearchTyp
e.DEFAULT)
.setQuery(querystring.field("Description")).setExplain(true).setSize(100).e
xecute().actionGet();

and previoulsy i was not giving analyzer name for searching.

I have read following documenthttp://www.elasticsearch.org/guide/reference/mapping/root-object-type...
that say that defining the mapping there is no need to define
explicit mapping for both cases index and search, and I have applied
this scheme. but my case it's working in indexing but not working on
searching. I needed to explicitly define search analyzer when i
searched for the given problem in this post i.e. I'avion .

What could be the reason that mapping is not applied for searching?

Thanks

On Feb 7, 4:45 pm, Shay Banon <kim...@gmail.com (http://gmail.com)> wrote:

Here is a sample of where its working:gist:1759303 · GitHub.

On Tuesday, February 7, 2012 at 6:54 AM, sam wrote:

Thanks for reply.

Mapping :
{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

Here I have putting mapping to configured the index_analyzer and
search_analyzer,so it is not necessary to explicit define analyzer
when doing a search.
In my case which i pointed out in above question i have used
index_analyzer and search_analyzer both are same when index is
created.for searching i have used following search query:

QueryStringQueryBuilder querystring = new
QueryStringQueryBuilder(searchData);
SearchResponse searchResponse =
client.prepareSearch(indexName).setTypes(indexType).setSearchType(SearchTyp e.DEFAULT)
.setQuery(querystring.field("Description")).setExplain(true).setSize(100).e xecute().actionGet();

Thanks

On Feb 5, 4:30 pm, Shay Banon <kim...@gmail.com (http://gmail.com)> wrote:

If the search analyzer is configured with the same elision filter, then it will be applied when searching as well, but only on queries that do analysis on the search text (query_string, text, field). Which query did you execute?

On Friday, February 3, 2012 at 8:22 AM, sam wrote:

Hi

I have applied Elision Token Filter while indexing and searching,i
created an index for “l’avion” and searched for "avion" and got the
hits as “l’avion”.
this is correct and i believe that analyzer are done correctly for
Elision.
but when i search for “l’avion” i got zero hits.i was expecting one
hits.because i am searching the same which i indexed.

why i couldn't get the hit? if i have misunderstood the concept then
make it correct.

used custom analyzer settings :

{
"index" : {
"analysis" : {
"analyzer" : {
"manualindexanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
},
"manualsearchanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
}
},
"filter" : {
"myelision":{
"type" : "elision",
"articles" :["l", "m", "t", "qu", "n", "s", "j"]
}
}
}
}
}

Mapping :

{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

to put mapping i have used following java code:

PutMappingResponse actionGet =
client.admin().indices().preparePutMapping(indexName).setType(indexType).se tSource(analyzerMapping).execute().actionGet();
boolean acknowledged = actionGet.getAcknowledged();
logger.debug("put mapping acknowledged::" + acknowledged);

Thanks