Elision Token Filter with custom analyzer?


(Sam-4) #1

Hi

I have applied Elision Token Filter while indexing and searching,i
created an index for “l’avion” and searched for "avion" and got the
hits as “l’avion”.
this is correct and i believe that analyzer are done correctly for
Elision.
but when i search for “l’avion” i got zero hits.i was expecting one
hits.because i am searching the same which i indexed.

why i couldn't get the hit? if i have misunderstood the concept then
make it correct.

used custom analyzer settings :

{
"index" : {
"analysis" : {
"analyzer" : {
"manualindexanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
},
"manualsearchanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
}
},
"filter" : {
"myelision":{
"type" : "elision",
"articles" :["l", "m", "t", "qu", "n", "s", "j"]
}
}
}
}
}

Mapping :

{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

to put mapping i have used following java code:

PutMappingResponse actionGet =
client.admin().indices().preparePutMapping(indexName).setType(indexType).setSource(analyzerMapping).execute().actionGet();
boolean acknowledged = actionGet.getAcknowledged();
logger.debug("put mapping acknowledged::" + acknowledged);

Thanks


(Shay Banon) #2

If the search analyzer is configured with the same elision filter, then it will be applied when searching as well, but only on queries that do analysis on the search text (query_string, text, field). Which query did you execute?

On Friday, February 3, 2012 at 8:22 AM, sam wrote:

Hi

I have applied Elision Token Filter while indexing and searching,i
created an index for “l’avion” and searched for "avion" and got the
hits as “l’avion”.
this is correct and i believe that analyzer are done correctly for
Elision.
but when i search for “l’avion” i got zero hits.i was expecting one
hits.because i am searching the same which i indexed.

why i couldn't get the hit? if i have misunderstood the concept then
make it correct.

used custom analyzer settings :

{
"index" : {
"analysis" : {
"analyzer" : {
"manualindexanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
},
"manualsearchanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
}
},
"filter" : {
"myelision":{
"type" : "elision",
"articles" :["l", "m", "t", "qu", "n", "s", "j"]
}
}
}
}
}

Mapping :

{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

to put mapping i have used following java code:

PutMappingResponse actionGet =
client.admin().indices().preparePutMapping(indexName).setType(indexType).setSource(analyzerMapping).execute().actionGet();
boolean acknowledged = actionGet.getAcknowledged();
logger.debug("put mapping acknowledged::" + acknowledged);

Thanks


(Sam-4) #3

Thanks for reply.

Mapping :
{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

Here I have putting mapping to configured the index_analyzer and
search_analyzer,so it is not necessary to explicit define analyzer
when doing a search.
In my case which i pointed out in above question i have used
index_analyzer and search_analyzer both are same when index is
created.for searching i have used following search query:

         QueryStringQueryBuilder querystring = new

QueryStringQueryBuilder(searchData);
SearchResponse searchResponse =
client.prepareSearch(indexName).setTypes(indexType).setSearchType(SearchType.DEFAULT)
.setQuery(querystring.field("Description")).setExplain(true).setSize(100).execute().actionGet();

Thanks

On Feb 5, 4:30 pm, Shay Banon kim...@gmail.com wrote:

If the search analyzer is configured with the same elision filter, then it will be applied when searching as well, but only on queries that do analysis on the search text (query_string, text, field). Which query did you execute?

On Friday, February 3, 2012 at 8:22 AM, sam wrote:

Hi

I have applied Elision Token Filter while indexing and searching,i
created an index for “l’avion” and searched for "avion" and got the
hits as “l’avion”.
this is correct and i believe that analyzer are done correctly for
Elision.
but when i search for “l’avion” i got zero hits.i was expecting one
hits.because i am searching the same which i indexed.

why i couldn't get the hit? if i have misunderstood the concept then
make it correct.

used custom analyzer settings :

{
"index" : {
"analysis" : {
"analyzer" : {
"manualindexanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
},
"manualsearchanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
}
},
"filter" : {
"myelision":{
"type" : "elision",
"articles" :["l", "m", "t", "qu", "n", "s", "j"]
}
}
}
}
}

Mapping :

{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

to put mapping i have used following java code:

PutMappingResponse actionGet =
client.admin().indices().preparePutMapping(indexName).setType(indexType).se tSource(analyzerMapping).execute().actionGet();
boolean acknowledged = actionGet.getAcknowledged();
logger.debug("put mapping acknowledged::" + acknowledged);

Thanks


(Shay Banon) #4

Here is a sample of where its working: https://gist.github.com/1759303.

On Tuesday, February 7, 2012 at 6:54 AM, sam wrote:

Thanks for reply.

Mapping :
{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

Here I have putting mapping to configured the index_analyzer and
search_analyzer,so it is not necessary to explicit define analyzer
when doing a search.
In my case which i pointed out in above question i have used
index_analyzer and search_analyzer both are same when index is
created.for searching i have used following search query:

QueryStringQueryBuilder querystring = new
QueryStringQueryBuilder(searchData);
SearchResponse searchResponse =
client.prepareSearch(indexName).setTypes(indexType).setSearchType(SearchType.DEFAULT)
.setQuery(querystring.field("Description")).setExplain(true).setSize(100).execute().actionGet();

Thanks

On Feb 5, 4:30 pm, Shay Banon <kim...@gmail.com (http://gmail.com)> wrote:

If the search analyzer is configured with the same elision filter, then it will be applied when searching as well, but only on queries that do analysis on the search text (query_string, text, field). Which query did you execute?

On Friday, February 3, 2012 at 8:22 AM, sam wrote:

Hi

I have applied Elision Token Filter while indexing and searching,i
created an index for “l’avion” and searched for "avion" and got the
hits as “l’avion”.
this is correct and i believe that analyzer are done correctly for
Elision.
but when i search for “l’avion” i got zero hits.i was expecting one
hits.because i am searching the same which i indexed.

why i couldn't get the hit? if i have misunderstood the concept then
make it correct.

used custom analyzer settings :

{
"index" : {
"analysis" : {
"analyzer" : {
"manualindexanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
},
"manualsearchanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
}
},
"filter" : {
"myelision":{
"type" : "elision",
"articles" :["l", "m", "t", "qu", "n", "s", "j"]
}
}
}
}
}

Mapping :

{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

to put mapping i have used following java code:

PutMappingResponse actionGet =
client.admin().indices().preparePutMapping(indexName).setType(indexType).se tSource(analyzerMapping).execute().actionGet();
boolean acknowledged = actionGet.getAcknowledged();
logger.debug("put mapping acknowledged::" + acknowledged);

Thanks


(Sam-4) #5

Thanks for reply.yes it's working. I figured out the problem, i added
logical analyzer name for searching, below is the used code:

QueryStringQueryBuilder querystring = new
QueryStringQueryBuilder(searchData).analyzer("manualsearchanalyzer");
SearchResponse searchResponse =
client.prepareSearch(indexName).setTypes(indexType).setSearchType(SearchTyp
e.DEFAULT)
.setQuery(querystring.field("Description")).setExplain(true).setSize(100).e
xecute().actionGet();

and previoulsy i was not giving analyzer name for searching.

I have read following document http://www.elasticsearch.org/guide/reference/mapping/root-object-type.html
that say that defining the mapping there is no need to define
explicit mapping for both cases index and search, and I have applied
this scheme. but my case it's working in indexing but not working on
searching. I needed to explicitly define search analyzer when i
searched for the given problem in this post i.e. I'avion .

What could be the reason that mapping is not applied for searching?

Thanks

On Feb 7, 4:45 pm, Shay Banon kim...@gmail.com wrote:

Here is a sample of where its working:https://gist.github.com/1759303.

On Tuesday, February 7, 2012 at 6:54 AM, sam wrote:

Thanks for reply.

Mapping :
{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

Here I have putting mapping to configured the index_analyzer and
search_analyzer,so it is not necessary to explicit define analyzer
when doing a search.
In my case which i pointed out in above question i have used
index_analyzer and search_analyzer both are same when index is
created.for searching i have used following search query:

QueryStringQueryBuilder querystring = new
QueryStringQueryBuilder(searchData);
SearchResponse searchResponse =
client.prepareSearch(indexName).setTypes(indexType).setSearchType(SearchTyp e.DEFAULT)
.setQuery(querystring.field("Description")).setExplain(true).setSize(100).e xecute().actionGet();

Thanks

On Feb 5, 4:30 pm, Shay Banon <kim...@gmail.com (http://gmail.com)> wrote:

If the search analyzer is configured with the same elision filter, then it will be applied when searching as well, but only on queries that do analysis on the search text (query_string, text, field). Which query did you execute?

On Friday, February 3, 2012 at 8:22 AM, sam wrote:

Hi

I have applied Elision Token Filter while indexing and searching,i
created an index for “l’avion” and searched for "avion" and got the
hits as “l’avion”.
this is correct and i believe that analyzer are done correctly for
Elision.
but when i search for “l’avion” i got zero hits.i was expecting one
hits.because i am searching the same which i indexed.

why i couldn't get the hit? if i have misunderstood the concept then
make it correct.

used custom analyzer settings :

{
"index" : {
"analysis" : {
"analyzer" : {
"manualindexanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
},
"manualsearchanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
}
},
"filter" : {
"myelision":{
"type" : "elision",
"articles" :["l", "m", "t", "qu", "n", "s", "j"]
}
}
}
}
}

Mapping :

{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

to put mapping i have used following java code:

PutMappingResponse actionGet =
client.admin().indices().preparePutMapping(indexName).setType(indexType).se tSource(analyzerMapping).execute().actionGet();
boolean acknowledged = actionGet.getAcknowledged();
logger.debug("put mapping acknowledged::" + acknowledged);

Thanks


(Shay Banon) #6

So now the code you posted is not the one you really used? This is annoying. If you want, I can answer the question you posted, but create a gist similar to what I created that exhibits what you expect, and I can have a look at it.

On Thursday, February 9, 2012 at 7:43 AM, sam wrote:

Thanks for reply.yes it's working. I figured out the problem, i added
logical analyzer name for searching, below is the used code:

QueryStringQueryBuilder querystring = new
QueryStringQueryBuilder(searchData).analyzer("manualsearchanalyzer");
SearchResponse searchResponse =
client.prepareSearch(indexName).setTypes(indexType).setSearchType(SearchTyp
e.DEFAULT)
.setQuery(querystring.field("Description")).setExplain(true).setSize(100).e
xecute().actionGet();

and previoulsy i was not giving analyzer name for searching.

I have read following document http://www.elasticsearch.org/guide/reference/mapping/root-object-type.html
that say that defining the mapping there is no need to define
explicit mapping for both cases index and search, and I have applied
this scheme. but my case it's working in indexing but not working on
searching. I needed to explicitly define search analyzer when i
searched for the given problem in this post i.e. I'avion .

What could be the reason that mapping is not applied for searching?

Thanks

On Feb 7, 4:45 pm, Shay Banon <kim...@gmail.com (http://gmail.com)> wrote:

Here is a sample of where its working:https://gist.github.com/1759303.

On Tuesday, February 7, 2012 at 6:54 AM, sam wrote:

Thanks for reply.

Mapping :
{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

Here I have putting mapping to configured the index_analyzer and
search_analyzer,so it is not necessary to explicit define analyzer
when doing a search.
In my case which i pointed out in above question i have used
index_analyzer and search_analyzer both are same when index is
created.for searching i have used following search query:

QueryStringQueryBuilder querystring = new
QueryStringQueryBuilder(searchData);
SearchResponse searchResponse =
client.prepareSearch(indexName).setTypes(indexType).setSearchType(SearchTyp e.DEFAULT)
.setQuery(querystring.field("Description")).setExplain(true).setSize(100).e xecute().actionGet();

Thanks

On Feb 5, 4:30 pm, Shay Banon <kim...@gmail.com (http://gmail.com)> wrote:

If the search analyzer is configured with the same elision filter, then it will be applied when searching as well, but only on queries that do analysis on the search text (query_string, text, field). Which query did you execute?

On Friday, February 3, 2012 at 8:22 AM, sam wrote:

Hi

I have applied Elision Token Filter while indexing and searching,i
created an index for “l’avion” and searched for "avion" and got the
hits as “l’avion”.
this is correct and i believe that analyzer are done correctly for
Elision.
but when i search for “l’avion” i got zero hits.i was expecting one
hits.because i am searching the same which i indexed.

why i couldn't get the hit? if i have misunderstood the concept then
make it correct.

used custom analyzer settings :

{
"index" : {
"analysis" : {
"analyzer" : {
"manualindexanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
},
"manualsearchanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
}
},
"filter" : {
"myelision":{
"type" : "elision",
"articles" :["l", "m", "t", "qu", "n", "s", "j"]
}
}
}
}
}

Mapping :

{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

to put mapping i have used following java code:

PutMappingResponse actionGet =
client.admin().indices().preparePutMapping(indexName).setType(indexType).se tSource(analyzerMapping).execute().actionGet();
boolean acknowledged = actionGet.getAcknowledged();
logger.debug("put mapping acknowledged::" + acknowledged);

Thanks


(Sam-4) #7

sorry for inconvenient.i figured out the problem, i added logical
analyzer name for searching,i have following gist :

and previoulsy i was not giving analyzer name for searching.
I have read following document http://www.elasticsearch.org/guide/reference/mapping/root-object-type...
that say that defining the mapping there is no need to define
explicit mapping for both cases index and search, and I have applied
this scheme. but my case it's working in indexing but not working on
searching. I needed to explicitly define search analyzer when i
searched for the given problem in this post i.e. I'avion .
What could be the reason that mapping is not applied for searching?

Thanks

On Feb 9, 1:29 pm, Shay Banon kim...@gmail.com wrote:

So now the code you posted is not the one you really used? This is annoying. If you want, I can answer the question you posted, but create a gist similar to what I created that exhibits what you expect, and I can have a look at it.

On Thursday, February 9, 2012 at 7:43 AM, sam wrote:

Thanks for reply.yes it's working. I figured out the problem, i added
logical analyzer name for searching, below is the used code:

QueryStringQueryBuilder querystring = new
QueryStringQueryBuilder(searchData).analyzer("manualsearchanalyzer");
SearchResponse searchResponse =
client.prepareSearch(indexName).setTypes(indexType).setSearchType(SearchTyp
e.DEFAULT)
.setQuery(querystring.field("Description")).setExplain(true).setSize(100).e
xecute().actionGet();

and previoulsy i was not giving analyzer name for searching.

I have read following documenthttp://www.elasticsearch.org/guide/reference/mapping/root-object-type...
that say that defining the mapping there is no need to define
explicit mapping for both cases index and search, and I have applied
this scheme. but my case it's working in indexing but not working on
searching. I needed to explicitly define search analyzer when i
searched for the given problem in this post i.e. I'avion .

What could be the reason that mapping is not applied for searching?

Thanks

On Feb 7, 4:45 pm, Shay Banon <kim...@gmail.com (http://gmail.com)> wrote:

Here is a sample of where its working:https://gist.github.com/1759303.

On Tuesday, February 7, 2012 at 6:54 AM, sam wrote:

Thanks for reply.

Mapping :
{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

Here I have putting mapping to configured the index_analyzer and
search_analyzer,so it is not necessary to explicit define analyzer
when doing a search.
In my case which i pointed out in above question i have used
index_analyzer and search_analyzer both are same when index is
created.for searching i have used following search query:

QueryStringQueryBuilder querystring = new
QueryStringQueryBuilder(searchData);
SearchResponse searchResponse =
client.prepareSearch(indexName).setTypes(indexType).setSearchType(SearchTyp e.DEFAULT)
.setQuery(querystring.field("Description")).setExplain(true).setSize(100).e xecute().actionGet();

Thanks

On Feb 5, 4:30 pm, Shay Banon <kim...@gmail.com (http://gmail.com)> wrote:

If the search analyzer is configured with the same elision filter, then it will be applied when searching as well, but only on queries that do analysis on the search text (query_string, text, field). Which query did you execute?

On Friday, February 3, 2012 at 8:22 AM, sam wrote:

Hi

I have applied Elision Token Filter while indexing and searching,i
created an index for “l’avion” and searched for "avion" and got the
hits as “l’avion”.
this is correct and i believe that analyzer are done correctly for
Elision.
but when i search for “l’avion” i got zero hits.i was expecting one
hits.because i am searching the same which i indexed.

why i couldn't get the hit? if i have misunderstood the concept then
make it correct.

used custom analyzer settings :

{
"index" : {
"analysis" : {
"analyzer" : {
"manualindexanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
},
"manualsearchanalyzer" : {
"type":"custom",
"tokenizer" : "whitespace",
"filter" : ["lowercase","asciifolding","length","myelision"],
"char_filter" :["html_strip"]
}
},
"filter" : {
"myelision":{
"type" : "elision",
"articles" :["l", "m", "t", "qu", "n", "s", "j"]
}
}
}
}
}

Mapping :

{
"test_type" : {
"index_analyzer" : "manualindexanalyzer",
"search_analyzer" : "manualsearchanalyzer"
}
}

to put mapping i have used following java code:

PutMappingResponse actionGet =
client.admin().indices().preparePutMapping(indexName).setType(indexType).se tSource(analyzerMapping).execute().actionGet();
boolean acknowledged = actionGet.getAcknowledged();
logger.debug("put mapping acknowledged::" + acknowledged);

Thanks


(system) #8