# Protwords support?
**URL:** https://discuss.elastic.co/t/protwords-support/7480
**Category:** Elasticsearch
**Created:** [April 26, 2012, 5:23pm UTC](https://discuss.elastic.co/t/protwords-support/7480 "2012-04-26T17:23:26Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [April 26, 2012, 5:23pm UTC](https://discuss.elastic.co/t/protwords-support/7480/1 "2012-04-26T17:23:26Z")
Is there any support for Solr's protwords (stemming overrides)? If
not, should be doable to modify the existing stemmer filter to provide
a list of words. Never looked up the Solr's implementation.
--
Ivan
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [April 27, 2012, 10:28am UTC](https://discuss.elastic.co/t/protwords-support/7480/2 "2012-04-27T10:28:04Z")
Which analyzer are you after? Most stemming based ones support
stem\_exclusion setting.
On Thu, Apr 26, 2012 at 8:23 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> Is there any support for Solr's protwords (stemming overrides)? If
> not, should be doable to modify the existing stemmer filter to provide
> a list of words. Never looked up the Solr's implementation.
>
> --
> Ivan
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [April 27, 2012, 4:52pm UTC](https://discuss.elastic.co/t/protwords-support/7480/3 "2012-04-27T16:52:43Z")
Thanks! Never seen the stem\_exclusion property. Currently looking to
translate an existing custom Lucene analyzer to ES.
Do only the language analyzers support stem\_exclusion? It appears so.
It would be nice to use a stemmer filter directly with exclusions.
File-based, similar to stop words, would also be ideal. I can provide
a patch if it makes sense.
Cheers,
Ivan
On Fri, Apr 27, 2012 at 3:28 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> Which analyzer are you after? Most stemming based ones support
> stem\_exclusion setting.
>
> On Thu, Apr 26, 2012 at 8:23 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
>
> > Is there any support for Solr's protwords (stemming overrides)? If
> > not, should be doable to modify the existing stemmer filter to provide
> > a list of words. Never looked up the Solr's implementation.
> >
> > --
> > Ivan
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [April 29, 2012, 4:54pm UTC](https://discuss.elastic.co/t/protwords-support/7480/4 "2012-04-29T16:54:36Z")
The way that it works is when providing a stem exclusion,
a KeywordMarkerFilter is added with a hte provided list, then
the SnowballFilter ignores any token marked as keywork. We can add support
for KeywordMarkerFilter to allow for custom exclusion, but it will only
make sense with SnowballFilter afterwards (which is created with teh
stemmer filter in ES).
On Fri, Apr 27, 2012 at 7:52 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> Thanks! Never seen the stem\_exclusion property. Currently looking to
> translate an existing custom Lucene analyzer to ES.
>
> Do only the language analyzers support stem\_exclusion? It appears so.
> It would be nice to use a stemmer filter directly with exclusions.
> File-based, similar to stop words, would also be ideal. I can provide
> a patch if it makes sense.
>
> Cheers,
>
> Ivan
>
> On Fri, Apr 27, 2012 at 3:28 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
>
> > Which analyzer are you after? Most stemming based ones support
> > stem\_exclusion setting.
> >
> > On Thu, Apr 26, 2012 at 8:23 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> >
> > > Is there any support for Solr's protwords (stemming overrides)? If
> > > not, should be doable to modify the existing stemmer filter to provide
> > > a list of words. Never looked up the Solr's implementation.
> > >
> > > --
> > > Ivan
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [June 4, 2012, 11:11pm UTC](https://discuss.elastic.co/t/protwords-support/7480/5 "2012-06-04T23:11:13Z")
Finally getting a change to revisit this issue.
From what I can tell, it is not possible to modify an existing
analyzer, correct? Something like:
index :
analysis :
analyzer :
foobar :
type : english
tokenizer : mytokenizer
tokenizer :
mytokenizer : ...
In that case, I would need to create a custom analyzer with a stemmer
token filter with exclusions. Should I open an issue?
Cheers,
Ivan
On Sun, Apr 29, 2012 at 9:54 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> The way that it works is when providing a stem exclusion,
> a KeywordMarkerFilter is added with a hte provided list, then
> the SnowballFilter ignores any token marked as keywork. We can add support
> for KeywordMarkerFilter to allow for custom exclusion, but it will only make
> sense with SnowballFilter afterwards (which is created with teh stemmer
> filter in ES).
>
> On Fri, Apr 27, 2012 at 7:52 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
>
> > Thanks! Never seen the stem\_exclusion property. Currently looking to
> > translate an existing custom Lucene analyzer to ES.
> >
> > Do only the language analyzers support stem\_exclusion? It appears so.
> > It would be nice to use a stemmer filter directly with exclusions.
> > File-based, similar to stop words, would also be ideal. I can provide
> > a patch if it makes sense.
> >
> > Cheers,
> >
> > Ivan
> >
> > On Fri, Apr 27, 2012 at 3:28 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> >
> > > Which analyzer are you after? Most stemming based ones support
> > > stem\_exclusion setting.
> > >
> > > On Thu, Apr 26, 2012 at 8:23 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > >
> > > > Is there any support for Solr's protwords (stemming overrides)? If
> > > > not, should be doable to modify the existing stemmer filter to provide
> > > > a list of words. Never looked up the Solr's implementation.
> > > >
> > > > --
> > > > Ivan
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [June 8, 2012, 10:10pm UTC](https://discuss.elastic.co/t/protwords-support/7480/6 "2012-06-08T22:10:17Z")
When you set a tokenizer, you effectively set your own analyzer, so "type"
set to english is not really meaningful. I lost you a bit with what you are
after...
On Tue, Jun 5, 2012 at 1:11 AM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> Finally getting a change to revisit this issue.
>
> From what I can tell, it is not possible to modify an existing
> analyzer, correct? Something like:
>
> index :
> analysis :
> analyzer :
> foobar :
> type : english
> tokenizer : mytokenizer
> tokenizer :
> mytokenizer : ...
>
> In that case, I would need to create a custom analyzer with a stemmer
> token filter with exclusions. Should I open an issue?
>
> Cheers,
>
> Ivan
>
> On Sun, Apr 29, 2012 at 9:54 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
>
> > The way that it works is when providing a stem exclusion,
> > a KeywordMarkerFilter is added with a hte provided list, then
> > the SnowballFilter ignores any token marked as keywork. We can add
> > support
> > for KeywordMarkerFilter to allow for custom exclusion, but it will only
> > make
> > sense with SnowballFilter afterwards (which is created with teh stemmer
> > filter in ES).
> >
> > On Fri, Apr 27, 2012 at 7:52 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> >
> > > Thanks! Never seen the stem\_exclusion property. Currently looking to
> > > translate an existing custom Lucene analyzer to ES.
> > >
> > > Do only the language analyzers support stem\_exclusion? It appears so.
> > > It would be nice to use a stemmer filter directly with exclusions.
> > > File-based, similar to stop words, would also be ideal. I can provide
> > > a patch if it makes sense.
> > >
> > > Cheers,
> > >
> > > Ivan
> > >
> > > On Fri, Apr 27, 2012 at 3:28 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> > >
> > > > Which analyzer are you after? Most stemming based ones support
> > > > stem\_exclusion setting.
> > > >
> > > > On Thu, Apr 26, 2012 at 8:23 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > >
> > > > > Is there any support for Solr's protwords (stemming overrides)? If
> > > > > not, should be doable to modify the existing stemmer filter to
> > > > > provide
> > > > > a list of words. Never looked up the Solr's implementation.
> > > > >
> > > > > --
> > > > > Ivan
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [June 11, 2012, 6:06pm UTC](https://discuss.elastic.co/t/protwords-support/7480/7 "2012-06-11T18:06:48Z")
Sorry for being obtuse. The ultimate goal is to have stem\_exclusions
on custom analyzers. Currently, this feature is only supported by
specific language analyzers.
Your previous suggestion for having a KeywordMarkerFilter with custom
exclusions is a good one. Yes, it's placement in the filter list is
important, but most of Lucene currently works like this anyways.
Reverse the order of filters, and all of a sudden code doesn't work!
🙂
My later point about creating analyzers which are not custom was
merely a question of syntax. It seems trivial to simply "add on" to an
existing analyzer, but it was unclear whether or not it would work. I
guess the answer is that it does not.
Cheers,
Ivan
On Fri, Jun 8, 2012 at 3:10 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> When you set a tokenizer, you effectively set your own analyzer, so "type"
> set to english is not really meaningful. I lost you a bit with what you are
> after...
>
> On Tue, Jun 5, 2012 at 1:11 AM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
>
> > Finally getting a change to revisit this issue.
> >
> > From what I can tell, it is not possible to modify an existing
> > analyzer, correct? Something like:
> >
> > index :
> > analysis :
> > analyzer :
> > foobar :
> > type : english
> > tokenizer : mytokenizer
> > tokenizer :
> > mytokenizer : ...
> >
> > In that case, I would need to create a custom analyzer with a stemmer
> > token filter with exclusions. Should I open an issue?
> >
> > Cheers,
> >
> > Ivan
> >
> > On Sun, Apr 29, 2012 at 9:54 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> >
> > > The way that it works is when providing a stem exclusion,
> > > a KeywordMarkerFilter is added with a hte provided list, then
> > > the SnowballFilter ignores any token marked as keywork. We can add
> > > support
> > > for KeywordMarkerFilter to allow for custom exclusion, but it will only
> > > make
> > > sense with SnowballFilter afterwards (which is created with teh stemmer
> > > filter in ES).
> > >
> > > On Fri, Apr 27, 2012 at 7:52 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > >
> > > > Thanks! Never seen the stem\_exclusion property. Currently looking to
> > > > translate an existing custom Lucene analyzer to ES.
> > > >
> > > > Do only the language analyzers support stem\_exclusion? It appears so.
> > > > It would be nice to use a stemmer filter directly with exclusions.
> > > > File-based, similar to stop words, would also be ideal. I can provide
> > > > a patch if it makes sense.
> > > >
> > > > Cheers,
> > > >
> > > > Ivan
> > > >
> > > > On Fri, Apr 27, 2012 at 3:28 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> > > >
> > > > > Which analyzer are you after? Most stemming based ones support
> > > > > stem\_exclusion setting.
> > > > >
> > > > > On Thu, Apr 26, 2012 at 8:23 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > > >
> > > > > > Is there any support for Solr's protwords (stemming overrides)? If
> > > > > > not, should be doable to modify the existing stemmer filter to
> > > > > > provide
> > > > > > a list of words. Never looked up the Solr's implementation.
> > > > > >
> > > > > > --
> > > > > > Ivan
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [June 13, 2012, 6:14pm UTC](https://discuss.elastic.co/t/protwords-support/7480/8 "2012-06-13T18:14:51Z")
Yea, you can't do an "addon" for an existing analyzer, you need to fully
define the custom one. Regarding the filter, sounds good...
On Mon, Jun 11, 2012 at 8:06 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> Sorry for being obtuse. The ultimate goal is to have stem\_exclusions
> on custom analyzers. Currently, this feature is only supported by
> specific language analyzers.
>
> Your previous suggestion for having a KeywordMarkerFilter with custom
> exclusions is a good one. Yes, it's placement in the filter list is
> important, but most of Lucene currently works like this anyways.
> Reverse the order of filters, and all of a sudden code doesn't work!
> 🙂
>
> My later point about creating analyzers which are not custom was
> merely a question of syntax. It seems trivial to simply "add on" to an
> existing analyzer, but it was unclear whether or not it would work. I
> guess the answer is that it does not.
>
> Cheers,
>
> Ivan
>
> On Fri, Jun 8, 2012 at 3:10 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
>
> > When you set a tokenizer, you effectively set your own analyzer, so
> > "type"
> > set to english is not really meaningful. I lost you a bit with what you
> > are
> > after...
> >
> > On Tue, Jun 5, 2012 at 1:11 AM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> >
> > > Finally getting a change to revisit this issue.
> > >
> > > From what I can tell, it is not possible to modify an existing
> > > analyzer, correct? Something like:
> > >
> > > index :
> > > analysis :
> > > analyzer :
> > > foobar :
> > > type : english
> > > tokenizer : mytokenizer
> > > tokenizer :
> > > mytokenizer : ...
> > >
> > > In that case, I would need to create a custom analyzer with a stemmer
> > > token filter with exclusions. Should I open an issue?
> > >
> > > Cheers,
> > >
> > > Ivan
> > >
> > > On Sun, Apr 29, 2012 at 9:54 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> > >
> > > > The way that it works is when providing a stem exclusion,
> > > > a KeywordMarkerFilter is added with a hte provided list, then
> > > > the SnowballFilter ignores any token marked as keywork. We can add
> > > > support
> > > > for KeywordMarkerFilter to allow for custom exclusion, but it will
> > > > only
> > > > make
> > > > sense with SnowballFilter afterwards (which is created with teh
> > > > stemmer
> > > > filter in ES).
> > > >
> > > > On Fri, Apr 27, 2012 at 7:52 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > >
> > > > > Thanks! Never seen the stem\_exclusion property. Currently looking to
> > > > > translate an existing custom Lucene analyzer to ES.
> > > > >
> > > > > Do only the language analyzers support stem\_exclusion? It appears so.
> > > > > It would be nice to use a stemmer filter directly with exclusions.
> > > > > File-based, similar to stop words, would also be ideal. I can
> > > > > provide
> > > > > a patch if it makes sense.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Ivan
> > > > >
> > > > > On Fri, Apr 27, 2012 at 3:28 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com)
> > > > > wrote:
> > > > >
> > > > > > Which analyzer are you after? Most stemming based ones support
> > > > > > stem\_exclusion setting.
> > > > > >
> > > > > > On Thu, Apr 26, 2012 at 8:23 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > wrote:
> > > > > >
> > > > > > > Is there any support for Solr's protwords (stemming overrides)? If
> > > > > > > not, should be doable to modify the existing stemmer filter to
> > > > > > > provide
> > > > > > > a list of words. Never looked up the Solr's implementation.
> > > > > > >
> > > > > > > --
> > > > > > > Ivan
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 3:24am UTC](https://discuss.elastic.co/t/protwords-support/7480/9 "2017-07-06T03:24:29Z")