Adding custom filters to query_strings

Slater_Victoroff_2 · August 5, 2013, 6:12pm

I'm currently dealing with a search corpus that was put together by a
number of people with poor spelling skills and I was looking for a way to
mitigate this and I think that the solution for a simple solution is a
phonetic analyzer. However, I'm having some trouble actually implementing
this on the query side. I already have a number of custom analyzers
specified for indexing, but for some reason I just can't figure it out for
queries. If at all possible I would just like to use the analyzer already
specified in my settings file, but I'm not sure if that's possible.

My settings file is here: https://gist.github.com/Slater-Victoroff/6156734

and right now my query string is this:

{'query': {'query_string': {'analyzer': 'transcript_analyzer',
'default_field': 'searchable_text',
'query': u'huge capacitor'}},
'suggest': {'searchable_text_suggestions': {'term': {'field': 'searchable_text',
'max_term_freq': '50',
'size': '2'},
'text': u'huge capacitor'}}}

It fails on the transcript analyzer right now which is reasonable since I'm not sure it can peek at the other analyzers I have defined, but when I try defining things more explicitly:

{'query': {'query_string': {'analyzer': 'custom_analyzer',
'default_field': 'searchable_text',
'query': u'huge capacitor'}},
'settings': {'analysis': {'analyzer': {'custom_analyzer': {'filters': ['asciifolding',
'custom_stemmer',
'custom_phonetic'],
'tokenizer': 'standard',
'type': 'custom'}},
'filter': {'custom_phonetic': {'Replace': 'false',
'encoder': 'doublemetaphone',
'type': 'phonetic'},
'custom_stemmer': {'name': 'minimal_english',
'type': 'stemmer'}}}},
'suggest': {'searchable_text_suggestions': {'term': {'field': 'searchable_text',
'max_term_freq': '50',
'size': '2'},
'text': u'huge capacitor'}}}

I have no better luck, and that previous example also returns a parse error on "custom_analyzer"

Are there any full examples I could look at for query configuration? I'm having a lot of trouble here and not finding much help in the documentation.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Ivan · August 5, 2013, 7:25pm

You cannot define analyzers at search time inside the request. The
analyzers are defined either inside the elasticsearch config file or inside
the index mapping.

--
Ivan

On Mon, Aug 5, 2013 at 11:12 AM, Slater Victoroff svictoroff@edx.orgwrote:

I'm currently dealing with a search corpus that was put together by a
number of people with poor spelling skills and I was looking for a way to
mitigate this and I think that the solution for a simple solution is a
phonetic analyzer. However, I'm having some trouble actually implementing
this on the query side. I already have a number of custom analyzers
specified for indexing, but for some reason I just can't figure it out for
queries. If at all possible I would just like to use the analyzer already
specified in my settings file, but I'm not sure if that's possible.

My settings file is here: ElasticSearch Settings file · GitHub

and right now my query string is this:

{'query': {'query_string': {'analyzer': 'transcript_analyzer',
'default_field': 'searchable_text',
'query': u'huge capacitor'}},
'suggest': {'searchable_text_suggestions': {'term': {'field': 'searchable_text',
'max_term_freq': '50',
'size': '2'},
'text': u'huge capacitor'}}}

It fails on the transcript analyzer right now which is reasonable since I'm not sure it can peek at the other analyzers I have defined, but when I try defining things more explicitly:

{'query': {'query_string': {'analyzer': 'custom_analyzer',
'default_field': 'searchable_text',
'query': u'huge capacitor'}},
'settings': {'analysis': {'analyzer': {'custom_analyzer': {'filters': ['asciifolding',
'custom_stemmer',
'custom_phonetic'],
'tokenizer': 'standard',
'type': 'custom'}},
'filter': {'custom_phonetic': {'Replace': 'false',
'encoder': 'doublemetaphone',
'type': 'phonetic'},
'custom_stemmer': {'name': 'minimal_english',
'type': 'stemmer'}}}},
'suggest': {'searchable_text_suggestions': {'term': {'field': 'searchable_text',
'max_term_freq': '50',
'size': '2'},
'text': u'huge capacitor'}}}

I have no better luck, and that previous example also returns a parse error on "custom_analyzer"

Are there any full examples I could look at for query configuration? I'm having a lot of trouble here and not finding much help in the documentation.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Slater_Victoroff_2 · August 5, 2013, 8:08pm

Is there any way to access the analyzers I already have in my settings file
(see the gist)? Because I thought that I had already included my analyzers,
but I couldn't access them from the query.

On Monday, August 5, 2013 3:25:59 PM UTC-4, Ivan Brusic wrote:

You cannot define analyzers at search time inside the request. The
analyzers are defined either inside the elasticsearch config file or inside
the index mapping.

--
Ivan

On Mon, Aug 5, 2013 at 11:12 AM, Slater Victoroff <svict...@edx.org<javascript:>

wrote:

I'm currently dealing with a search corpus that was put together by a
number of people with poor spelling skills and I was looking for a way to
mitigate this and I think that the solution for a simple solution is a
phonetic analyzer. However, I'm having some trouble actually implementing
this on the query side. I already have a number of custom analyzers
specified for indexing, but for some reason I just can't figure it out for
queries. If at all possible I would just like to use the analyzer already
specified in my settings file, but I'm not sure if that's possible.

My settings file is here:
ElasticSearch Settings file · GitHub

and right now my query string is this:

{'query': {'query_string': {'analyzer': 'transcript_analyzer',
'default_field': 'searchable_text',
'query': u'huge capacitor'}},
'suggest': {'searchable_text_suggestions': {'term': {'field': 'searchable_text',
'max_term_freq': '50',
'size': '2'},
'text': u'huge capacitor'}}}

It fails on the transcript analyzer right now which is reasonable since I'm not sure it can peek at the other analyzers I have defined, but when I try defining things more explicitly:

{'query': {'query_string': {'analyzer': 'custom_analyzer',
'default_field': 'searchable_text',
'query': u'huge capacitor'}},
'settings': {'analysis': {'analyzer': {'custom_analyzer': {'filters': ['asciifolding',
'custom_stemmer',
'custom_phonetic'],
'tokenizer': 'standard',
'type': 'custom'}},
'filter': {'custom_phonetic': {'Replace': 'false',
'encoder': 'doublemetaphone',
'type': 'phonetic'},
'custom_stemmer': {'name': 'minimal_english',
'type': 'stemmer'}}}},
'suggest': {'searchable_text_suggestions': {'term': {'field': 'searchable_text',
'max_term_freq': '50',
'size': '2'},
'text': u'huge capacitor'}}}

I have no better luck, and that previous example also returns a parse error on "custom_analyzer"

Are there any full examples I could look at for query configuration? I'm having a lot of trouble here and not finding much help in the documentation.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Ivan · August 5, 2013, 10:26pm

If the mapping was provided during the creation of the index then you can
use the Settings API:

If the analyzers were defined in elasticsearch.yml, then they are not
discoverable via any API AFAIK.

--
Ivan

On Mon, Aug 5, 2013 at 1:08 PM, Slater Victoroff svictoroff@edx.org wrote:

Is there any way to access the analyzers I already have in my settings
file (see the gist)? Because I thought that I had already included my
analyzers, but I couldn't access them from the query.

On Monday, August 5, 2013 3:25:59 PM UTC-4, Ivan Brusic wrote:

You cannot define analyzers at search time inside the request. The
analyzers are defined either inside the elasticsearch config file or inside
the index mapping.

--
Ivan

On Mon, Aug 5, 2013 at 11:12 AM, Slater Victoroff svict...@edx.orgwrote:

I'm currently dealing with a search corpus that was put together by a
number of people with poor spelling skills and I was looking for a way to
mitigate this and I think that the solution for a simple solution is a
phonetic analyzer. However, I'm having some trouble actually implementing
this on the query side. I already have a number of custom analyzers
specified for indexing, but for some reason I just can't figure it out for
queries. If at all possible I would just like to use the analyzer already
specified in my settings file, but I'm not sure if that's possible.

My settings file is here: https://gist.github.com/**
Slater-Victoroff/6156734https://gist.github.com/Slater-Victoroff/6156734

and right now my query string is this:

{'query': {'query_string': {'analyzer': 'transcript_analyzer',
'default_field': 'searchable_text',
'query': u'huge capacitor'}},
'suggest': {'searchable_text_suggestions'**: {'term': {'field': 'searchable_text',
'max_term_freq': '50',
'size': '2'},
'text': u'huge capacitor'}}}

It fails on the transcript analyzer right now which is reasonable since I'm not sure it can peek at the other analyzers I have defined, but when I try defining things more explicitly:

{'query': {'query_string': {'analyzer': 'custom_analyzer',
'default_field': 'searchable_text',
'query': u'huge capacitor'}},
'settings': {'analysis': {'analyzer': {'custom_analyzer': {'filters': ['asciifolding',
'custom_stemmer',
'custom_phonetic'],
'tokenizer': 'standard',
'type': 'custom'}},
'filter': {'custom_phonetic': {'Replace': 'false',
'encoder': 'doublemetaphone',
'type': 'phonetic'},
'custom_stemmer': {'name': 'minimal_english',
'type': 'stemmer'}}}},
'suggest': {'searchable_text_suggestions'**: {'term': {'field': 'searchable_text',
'max_term_freq': '50',
'size': '2'},
'text': u'huge capacitor'}}}

I have no better luck, and that previous example also returns a parse error on "custom_analyzer"

Are there any full examples I could look at for query configuration? I'm having a lot of trouble here and not finding much help in the documentation.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_out https://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.