# A less aggressive stemming token filter that strips only plural
**URL:** https://discuss.elastic.co/t/a-less-aggressive-stemming-token-filter-that-strips-only-plural/4033
**Category:** Elasticsearch
**Created:** [March 4, 2011, 6:05pm UTC](https://discuss.elastic.co/t/a-less-aggressive-stemming-token-filter-that-strips-only-plural/4033 "2011-03-04T18:05:11Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@Sorostaran](https://discuss.elastic.co/u/Sorostaran)
#### Post date: [March 4, 2011, 6:05pm UTC](https://discuss.elastic.co/t/a-less-aggressive-stemming-token-filter-that-strips-only-plural/4033/1 "2011-03-04T18:05:11Z")
Does anybody have a token filter (a cut-down Porter stemmer or something) that only does plural stemming in English for ElasticSearch? That seems like a common need for which few databases have out-of-the-box solutions. Any chance of adding ispell as a token filter?
---
### Author:  [@rmuir](https://discuss.elastic.co/u/rmuir)
#### Post date: [March 4, 2011, 6:47pm UTC](https://discuss.elastic.co/t/a-less-aggressive-stemming-token-filter-that-strips-only-plural/4033/2 "2011-03-04T18:47:56Z")
On Fri, Mar 4, 2011 at 1:05 PM, Sorostaran [steven@datafeedfile.com](mailto:steven@datafeedfile.com) wrote:
> Does anybody have a token filter (a cut-down Porter stemmer or something)
> that only does plural stemming in English for Elasticsearch? That seems like
> a common need for which few databases have out-of-the-box solutions. Any
> chance of adding ispell as a token filter?
Hi, in the upcoming lucene 3.1 there will be a variety of plural-only
and lighter implementations for at least common european languages,
you can see those here:
[http://svn.apache.org/repos/asf/lucene/dev/branches/branch\_3x/lucene/contrib/analyzers/common/src/java/org/apache/lucene/analysis/](http://svn.apache.org/repos/asf/lucene/dev/branches/branch_3x/lucene/contrib/analyzers/common/src/java/org/apache/lucene/analysis/)
additionally there is the capability to override all stemmers
(including these plural-only and lighter ones) e.g. by specifying
exceptions (that should be mapped to some special form:
[http://svn.apache.org/repos/asf/lucene/dev/branches/branch\_3x/lucene/contrib/analyzers/common/src/java/org/apache/lucene/analysis/miscellaneous/StemmerOverrideFilter.java](http://svn.apache.org/repos/asf/lucene/dev/branches/branch_3x/lucene/contrib/analyzers/common/src/java/org/apache/lucene/analysis/miscellaneous/StemmerOverrideFilter.java))
or words they should be ignored totally
[[http://svn.apache.org/repos/asf/lucene/dev/branches/branch\_3x/lucene/src/java/org/apache/lucene/analysis/KeywordMarkerFilter.java](http://svn.apache.org/repos/asf/lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/analysis/KeywordMarkerFilter.java)]
in my opinion this is the ideal way to go for many apps... start with
something very minimal like plural-only and add exceptions for stuff
that makes sense for your domain (e.g. "fatigues" is not the plural of
"fatigue" in english).
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [March 5, 2011, 6:58am UTC](https://discuss.elastic.co/t/a-less-aggressive-stemming-token-filter-that-strips-only-plural/4033/3 "2011-03-05T06:58:13Z")
Heya,
Thanks Robert!. Many of the specific language analyzers are already exposed in elasticsearch based on Lucene 3.0.3. Once 3.1 is released, all of these will be exposed as well.
-shay.banon
On Friday, March 4, 2011 at 8:47 PM, Robert Muir wrote:
> On Fri, Mar 4, 2011 at 1:05 PM, Sorostaran [steven@datafeedfile.com](mailto:steven@datafeedfile.com) wrote:
>
> > Does anybody have a token filter (a cut-down Porter stemmer or something)
> > that only does plural stemming in English for Elasticsearch? That seems like
> > a common need for which few databases have out-of-the-box solutions. Any
> > chance of adding ispell as a token filter?
>
> Hi, in the upcoming lucene 3.1 there will be a variety of plural-only
> and lighter implementations for at least common european languages,
> you can see those here:
> [http://svn.apache.org/repos/asf/lucene/dev/branches/branch\_3x/lucene/contrib/analyzers/common/src/java/org/apache/lucene/analysis/](http://svn.apache.org/repos/asf/lucene/dev/branches/branch_3x/lucene/contrib/analyzers/common/src/java/org/apache/lucene/analysis/)
>
> additionally there is the capability to override all stemmers
> (including these plural-only and lighter ones) e.g. by specifying
> exceptions (that should be mapped to some special form:
> [http://svn.apache.org/repos/asf/lucene/dev/branches/branch\_3x/lucene/contrib/analyzers/common/src/java/org/apache/lucene/analysis/miscellaneous/StemmerOverrideFilter.java](http://svn.apache.org/repos/asf/lucene/dev/branches/branch_3x/lucene/contrib/analyzers/common/src/java/org/apache/lucene/analysis/miscellaneous/StemmerOverrideFilter.java))
> or words they should be ignored totally
> [[http://svn.apache.org/repos/asf/lucene/dev/branches/branch\_3x/lucene/src/java/org/apache/lucene/analysis/KeywordMarkerFilter.java](http://svn.apache.org/repos/asf/lucene/dev/branches/branch_3x/lucene/src/java/org/apache/lucene/analysis/KeywordMarkerFilter.java)]
>
> in my opinion this is the ideal way to go for many apps... start with
> something very minimal like plural-only and add exceptions for stuff
> that makes sense for your domain (e.g. "fatigues" is not the plural of
> "fatigue" in english).
---
### Author:  [@rmuir](https://discuss.elastic.co/u/rmuir)
#### Post date: [March 5, 2011, 5:58pm UTC](https://discuss.elastic.co/t/a-less-aggressive-stemming-token-filter-that-strips-only-plural/4033/4 "2011-03-05T17:58:24Z")
On Sat, Mar 5, 2011 at 1:58 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> Heya,
> Thanks Robert!. Many of the specific language analyzers are already
> exposed in elasticsearch based on Lucene 3.0.3. Once 3.1 is released, all of
> these will be exposed as well.
> -shay.banon
Hey, one question. Do elasticsearch users "typically" use the lucene
Analyzer classes or do they construct them "on-the-fly" from
tokenstreams? (I think i've seen use of "custom" for this?)
The reason i say this is that most of the actual Analyzer classes just
use the "heavy-duty" snowball stuff... even if more reasonable
alternatives are available.
just wondering if in 3.2 it would be worth our effort to consider
improving these Analyzers, e.g. default them to less aggressive
stemmers... if they are being used for more than just examples 馃檪
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [March 6, 2011, 4:12am UTC](https://discuss.elastic.co/t/a-less-aggressive-stemming-token-filter-that-strips-only-plural/4033/5 "2011-03-06T04:12:33Z")
On Saturday, March 5, 2011 at 7:58 PM, Robert Muir wrote:
On Sat, Mar 5, 2011 at 1:58 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> > Heya,
> > Thanks Robert!. Many of the specific language analyzers are already
> > exposed in elasticsearch based on Lucene 3.0.3. Once 3.1 is released, all of
> > these will be exposed as well.
> > -shay.banon
>
> Hey, one question. Do elasticsearch users "typically" use the lucene
> Analyzer classes or do they construct them "on-the-fly" from
> tokenstreams? (I think i've seen use of "custom" for this?)
>
> The reason i say this is that most of the actual Analyzer classes just
> use the "heavy-duty" snowball stuff... even if more reasonable
> alternatives are available.
>
> just wondering if in 3.2 it would be worth our effort to consider
> improving these Analyzers, e.g. default them to less aggressive
> stemmers... if they are being used for more than just examples 馃檪
> Its got both, exposing the pre built analyzers that comes out of the box, and the ability to create custom ones (which include a tokenizers and one or more filters). Definitely, the "default" ones are probably more popular, since its much simpler to configure.
The default analyzers are certainly used for more than just examples, simply because its much simpler to use and not many users (initially) go to the depth of understanding and configuring their own analyzers. So, I would say a better out of the box analyzers would go a long way. It will certainly be hte case for pure Lucene, and possibly it can be done in elasticsearch by exposing more pre built analyzers.
>
---
### Author:  [@linsms](https://discuss.elastic.co/u/linsms)
#### Post date: [May 2, 2011, 12:18pm UTC](https://discuss.elastic.co/t/a-less-aggressive-stemming-token-filter-that-strips-only-plural/4033/6 "2011-05-02T12:18:35Z")
Hi,
I'm newbie in lucene (I'm using 3.1) and I'm trying to use SpanishAnalyzer
to make a query, but I get unspected results: I get cut querys with terms
like "despues" or "ciempies" (body:despu and body:ciempi).
I understand that with your explanation I can disable the plural's cut, but
I don't know how to do it.
Could you help me?
Thanks in advance.
--
View this message in context: [http://elasticsearch-users.115913.n3.nabble.com/A-less-aggressive-stemming-token-filter-that-strips-only-plural-tp2634846p2889629.html](http://elasticsearch-users.115913.n3.nabble.com/A-less-aggressive-stemming-token-filter-that-strips-only-plural-tp2634846p2889629.html)
Sent from the ElasticSearch Users mailing list archive at [Nabble.com](http://Nabble.com).
---
### Author:  [@otisg](https://discuss.elastic.co/u/otisg)
#### Post date: [May 5, 2011, 3:29pm UTC](https://discuss.elastic.co/t/a-less-aggressive-stemming-token-filter-that-strips-only-plural/4033/7 "2011-05-05T15:29:15Z")
linsms,
I think Robert just addressed this a day earlier in this same thread:
> **[Google Groups](https://groups.google.com/access-error?continue=https:%2F%2Fgroups.google.com%2Fa%2Felasticsearch.com%2Fg%2Fusers%2Fc%2Fl-slj1-73Gk)**
## Otis
Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch
Lucene ecosystem search :: [http://search-lucene.com/](http://search-lucene.com/)
On May 2, 8:17 am, linsms [lin...@gmail.com](mailto:lin...@gmail.com) wrote:
> Hi,
>
> I'm newbie in lucene (I'm using 3.1) and I'm trying to use SpanishAnalyzer
> to make a query, but I get unspected results: I get cut querys with terms
> like "despues" or "ciempies" (body:despu and body:ciempi).
>
> I understand that with your explanation I can disable the plural's cut, but
> I don't know how to do it.
>
> Could you help me?
>
> Thanks in advance.
>
> --
> View this message in context:[http://elasticsearch-users.115913.n3.nabble.com/A-less-aggressive-ste](http://elasticsearch-users.115913.n3.nabble.com/A-less-aggressive-ste)...
> Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com).
---
### Author:  [@rmuir](https://discuss.elastic.co/u/rmuir)
#### Post date: [May 5, 2011, 3:37pm UTC](https://discuss.elastic.co/t/a-less-aggressive-stemming-token-filter-that-strips-only-plural/4033/8 "2011-05-05T15:37:58Z")
On Thu, May 5, 2011 at 11:29 AM, Otis [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com) wrote:
> linsms,
>
> I think Robert just addressed this a day earlier in this same thread:
> [http://groups.google.com/a/elasticsearch.com/group/users/browse\_thread/thread/97eb258f5fbbdc69](http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/97eb258f5fbbdc69)
Thanks Otis, as far as elasticsearch, now that its on 3.1 one easy win
might be to expose factories for some of these filters (if not
already)?
In combination with factories for the \*LightStemFilter and
\*MinimalFilters, i would also recommend exposing the new
StemmerOverrideFilter and KeywordMarkerFilter.
This way, users can pick less aggressive algorithms and then tune any
exceptions to fit.
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [May 5, 2011, 6:05pm UTC](https://discuss.elastic.co/t/a-less-aggressive-stemming-token-filter-that-strips-only-plural/4033/9 "2011-05-05T18:05:34Z")
Heya Robert,
Thanks!, yes, I should expose those as well as built in options. Here is the issue: [Analysis: Expose light and minimal language token filters 路 Issue #908 路 elastic/elasticsearch 路 GitHub](https://github.com/elasticsearch/elasticsearch/issues/908/).
-shay.banon
On Thursday, May 5, 2011 at 6:37 PM, Robert Muir wrote:
> On Thu, May 5, 2011 at 11:29 AM, Otis [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com) wrote:
>
> > linsms,
> >
> > I think Robert just addressed this a day earlier in this same thread:
> > [http://groups.google.com/a/elasticsearch.com/group/users/browse\_thread/thread/97eb258f5fbbdc69](http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/97eb258f5fbbdc69)
>
> Thanks Otis, as far as elasticsearch, now that its on 3.1 one easy win
> might be to expose factories for some of these filters (if not
> already)?
> In combination with factories for the \*LightStemFilter and
> \*MinimalFilters, i would also recommend exposing the new
> StemmerOverrideFilter and KeywordMarkerFilter.
>
> This way, users can pick less aggressive algorithms and then tune any
> exceptions to fit.
---
### Author:  [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)
#### Post date: [May 5, 2011, 6:40pm UTC](https://discuss.elastic.co/t/a-less-aggressive-stemming-token-filter-that-strips-only-plural/4033/10 "2011-05-05T18:40:04Z")
This will be really useful, can't wait to see it available. 馃檪
On Thu, May 5, 2011 at 8:05 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:
> Heya Robert,
>
> Thanks!, yes, I should expose those as well as built in options. Here is
> the issue: [Analysis: Expose light and minimal language token filters 路 Issue #908 路 elastic/elasticsearch 路 GitHub](https://github.com/elasticsearch/elasticsearch/issues/908/).
>
> -shay.banon
>
> On Thursday, May 5, 2011 at 6:37 PM, Robert Muir wrote:
>
> On Thu, May 5, 2011 at 11:29 AM, Otis [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com) wrote:
>
> linsms,
>
> I think Robert just addressed this a day earlier in this same thread:
>
> [http://groups.google.com/a/elasticsearch.com/group/users/browse\_thread/thread/97eb258f5fbbdc69](http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/97eb258f5fbbdc69)
>
> Thanks Otis, as far as elasticsearch, now that its on 3.1 one easy win
> might be to expose factories for some of these filters (if not
> already)?
> In combination with factories for the \*LightStemFilter and
> \*MinimalFilters, i would also recommend exposing the new
> StemmerOverrideFilter and KeywordMarkerFilter.
>
> This way, users can pick less aggressive algorithms and then tune any
> exceptions to fit.
---
### Author:  [@tfreitas](https://discuss.elastic.co/u/tfreitas)
#### Post date: [May 5, 2011, 9:40pm UTC](https://discuss.elastic.co/t/a-less-aggressive-stemming-token-filter-that-strips-only-plural/4033/11 "2011-05-05T21:40:18Z")
Hi Shay
In Code
[https://github.com/elasticsearch/elasticsearch/blob/master/modules/elasticsearch/src/main/java/org/elasticsearch/index/analysis/SpanishAnalyzerProvider.java](https://github.com/elasticsearch/elasticsearch/blob/master/modules/elasticsearch/src/main/java/org/elasticsearch/index/analysis/SpanishAnalyzerProvider.java)
* * *
```
@Inject public SpanishAnalyzerProvider(Index index, @IndexSettings
```
## Settings indexSettings, @Assisted String name, @Assisted Settings settings) { super(index, indexSettings, name, settings); analyzer = new SpanishAnalyzer(version, Analysis.parseStopWords(settings, ArabicAnalyzer.getDefaultStopSet()), Analysis.parseStemExclusion(settings, CharArraySet.EMPTY\_SET)); }
ArabicAnalyzer.getDefaultStopSet() is Ok?
On May 5, 2:05 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> Heya Robert,
>
> Thanks!, yes, I should expose those as well as built in options. Here is the issue:[Analysis: Expose light and minimal language token filters 路 Issue #908 路 elastic/elasticsearch 路 GitHub](https://github.com/elasticsearch/elasticsearch/issues/908/).
>
> -shay.banon
>
> On Thursday, May 5, 2011 at 6:37 PM, Robert Muir wrote:
>
> > On Thu, May 5, 2011 at 11:29 AM, Otis [otis.gospodne...@gmail.com](mailto:otis.gospodne...@gmail.com) wrote:
> >
> > > linsms,
>
> > > I think Robert just addressed this a day earlier in this same thread:
> > > [http://groups.google.com/a/elasticsearch.com/group/users/browse\_threa](http://groups.google.com/a/elasticsearch.com/group/users/browse_threa)...
>
> > Thanks Otis, as far as elasticsearch, now that its on 3.1 one easy win
> > might be to expose factories for some of these filters (if not
> > already)?
> > In combination with factories for the \*LightStemFilter and
> > \*MinimalFilters, i would also recommend exposing the new
> > StemmerOverrideFilter and KeywordMarkerFilter.
>
> > This way, users can pick less aggressive algorithms and then tune any
> > exceptions to fit.
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [May 5, 2011, 9:47pm UTC](https://discuss.elastic.co/t/a-less-aggressive-stemming-token-filter-that-strips-only-plural/4033/12 "2011-05-05T21:47:22Z")
Ha, saw it as well while trying to add custom "handlers" for default lang stopwords to stop filter. Pushed a fix.
On Friday, May 6, 2011 at 12:40 AM, tfreitas wrote:
> Hi Shay
>
> In Code
> [https://github.com/elasticsearch/elasticsearch/blob/master/modules/elasticsearch/src/main/java/org/elasticsearch/index/analysis/SpanishAnalyzerProvider.java](https://github.com/elasticsearch/elasticsearch/blob/master/modules/elasticsearch/src/main/java/org/elasticsearch/index/analysis/SpanishAnalyzerProvider.java)
>
> * * *
>
> ## @Inject public SpanishAnalyzerProvider(Index index, @IndexSettings Settings indexSettings, @Assisted String name, @Assisted Settings settings) { super(index, indexSettings, name, settings); analyzer = new SpanishAnalyzer(version, Analysis.parseStopWords(settings, ArabicAnalyzer.getDefaultStopSet()), Analysis.parseStemExclusion(settings, CharArraySet.EMPTY\_SET)); }
>
> ArabicAnalyzer.getDefaultStopSet() is Ok?
>
> On May 5, 2:05 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
>
> > Heya Robert,
> >
> > Thanks!, yes, I should expose those as well as built in options. Here is the issue:[Analysis: Expose light and minimal language token filters 路 Issue #908 路 elastic/elasticsearch 路 GitHub](https://github.com/elasticsearch/elasticsearch/issues/908/).
> >
> > -shay.banon
> >
> > On Thursday, May 5, 2011 at 6:37 PM, Robert Muir wrote:
> >
> > > On Thu, May 5, 2011 at 11:29 AM, Otis [otis.gospodne...@gmail.com](mailto:otis.gospodne...@gmail.com) wrote:
> > >
> > > > linsms,
> >
> > > > I think Robert just addressed this a day earlier in this same thread:
> > > > [http://groups.google.com/a/elasticsearch.com/group/users/browse\_threa](http://groups.google.com/a/elasticsearch.com/group/users/browse_threa)...
> >
> > > Thanks Otis, as far as elasticsearch, now that its on 3.1 one easy win
> > > might be to expose factories for some of these filters (if not
> > > already)?
> > > In combination with factories for the \*LightStemFilter and
> > > \*MinimalFilters, i would also recommend exposing the new
> > > StemmerOverrideFilter and KeywordMarkerFilter.
> >
> > > This way, users can pick less aggressive algorithms and then tune any
> > > exceptions to fit.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:06am UTC](https://discuss.elastic.co/t/a-less-aggressive-stemming-token-filter-that-strips-only-plural/4033/13 "2017-07-06T04:06:50Z")