# ES Plugin to extend Lucene's Standard Tokenizer
**URL:** https://discuss.elastic.co/t/es-plugin-to-extend-lucenes-standard-tokenizer/19670
**Category:** Elasticsearch
**Created:** [September 9, 2014, 2:26am UTC](https://discuss.elastic.co/t/es-plugin-to-extend-lucenes-standard-tokenizer/19670 "2014-09-09T02:26:18Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@Bryan\_Warner](https://discuss.elastic.co/u/Bryan_Warner)
#### Post date: [September 9, 2014, 2:26am UTC](https://discuss.elastic.co/t/es-plugin-to-extend-lucenes-standard-tokenizer/19670/1 "2014-09-09T02:26:18Z")
Hi all,
Recently, I've been working on an extension to Lucene's Standard Tokenizer
that allows the user to customize / override the default word boundary
break rules for Unicode characters. The Standard Tokenizer implements the
word break rules from the Unicode Text segmentation
[http://www.unicode.org/reports/tr29/](http://www.unicode.org/reports/tr29/) algorithm where most punctuation
symbols (except for underscore '\_') are treated as hard word breaks (e.g.
"@foo" , "#foo" are tokenized to "foo"). While the Standard Tokenizer works
great in most cases, I found that being unable to override the default word
break rules was quite limiting especially since a lot of these punctuation
symbols have important meaning now on the web (@ - mentions, # - hashtags,
etc.)
I've wrapped this extension to the Standard Tokenizer in an ElasticSearch
plugin, which can be found at -
[https://github.com/bbguitar77/elasticsearch-analysis-standardext](https://github.com/bbguitar77/elasticsearch-analysis-standardext) ...
definitely looking for feedback as this is my first go at an ElasticSearch
plugin!
I'm hoping other ElasticSearch / Lucene users find this helpful.
Cheers!
Bryan
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/929dc7c3-ff99-43a4-a287-1a8f89d86e3f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/929dc7c3-ff99-43a4-a287-1a8f89d86e3f%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@vineeth\_mohan\_2](https://discuss.elastic.co/u/vineeth_mohan_2)
#### Post date: [September 9, 2014, 3:36am UTC](https://discuss.elastic.co/t/es-plugin-to-extend-lucenes-standard-tokenizer/19670/2 "2014-09-09T03:36:01Z")
Hello Bryan ,
Congrats on your first plugin.
I have a question here - Can you implement the whole plugin by using
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
tokenizer ?
Is your plugin providing any advantage over going this approach ?
Thanks
Vineeth
On Tue, Sep 9, 2014 at 7:56 AM, Bryan Warner [bryan.warner@gmail.com](mailto:bryan.warner@gmail.com) wrote:
> Hi all,
>
> Recently, I've been working on an extension to Lucene's Standard Tokenizer
> that allows the user to customize / override the default word boundary
> break rules for Unicode characters. The Standard Tokenizer implements the
> word break rules from the Unicode Text segmentation
> [http://www.unicode.org/reports/tr29/](http://www.unicode.org/reports/tr29/) algorithm where most punctuation
> symbols (except for underscore '\_') are treated as hard word breaks (e.g.
> "@foo" , "#foo" are tokenized to "foo"). While the Standard Tokenizer works
> great in most cases, I found that being unable to override the default word
> break rules was quite limiting especially since a lot of these punctuation
> symbols have important meaning now on the web (@ - mentions, # - hashtags,
> etc.)
>
> I've wrapped this extension to the Standard Tokenizer in an Elasticsearch
> plugin, which can be found at -
> [GitHub - bbguitar77/elasticsearch-analysis-standardext](https://github.com/bbguitar77/elasticsearch-analysis-standardext) ...
> definitely looking for feedback as this is my first go at an Elasticsearch
> plugin!
>
> I'm hoping other Elasticsearch / Lucene users find this helpful.
>
> Cheers!
> Bryan
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/929dc7c3-ff99-43a4-a287-1a8f89d86e3f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/929dc7c3-ff99-43a4-a287-1a8f89d86e3f%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/929dc7c3-ff99-43a4-a287-1a8f89d86e3f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/929dc7c3-ff99-43a4-a287-1a8f89d86e3f%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGdPd5koucm5\_kWjrVqs%3DEQm8fySnuM81i\_j3KGUKi3NafaPpQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGdPd5koucm5_kWjrVqs%3DEQm8fySnuM81i_j3KGUKi3NafaPpQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [September 9, 2014, 7:29am UTC](https://discuss.elastic.co/t/es-plugin-to-extend-lucenes-standard-tokenizer/19670/3 "2014-09-09T07:29:33Z")
Also congrats for writing a plugin!
For alternative punctuation tokenizing, you can also look at the "classic"
tokenizer, which is the behavior of Lucene standard tokenization before
3.1, when it switched to Unicode text segmentation.
Best,
Jörg
On Tue, Sep 9, 2014 at 5:36 AM, vineeth mohan [vm.vineethmohan@gmail.com](mailto:vm.vineethmohan@gmail.com)
wrote:
> Hello Bryan ,
>
> Congrats on your first plugin.
> I have a question here - Can you implement the whole plugin by using
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-pattern-tokenizer.html)
> tokenizer ?
>
> Is your plugin providing any advantage over going this approach ?
>
> Thanks
> Vineeth
>
> On Tue, Sep 9, 2014 at 7:56 AM, Bryan Warner [bryan.warner@gmail.com](mailto:bryan.warner@gmail.com)
> wrote:
>
> > Hi all,
> >
> > Recently, I've been working on an extension to Lucene's Standard
> > Tokenizer that allows the user to customize / override the default word
> > boundary break rules for Unicode characters. The Standard Tokenizer
> > implements the word break rules from the Unicode Text segmentation
> > [http://www.unicode.org/reports/tr29/](http://www.unicode.org/reports/tr29/) algorithm where most punctuation
> > symbols (except for underscore '\_') are treated as hard word breaks (e.g.
> > "@foo" , "#foo" are tokenized to "foo"). While the Standard Tokenizer works
> > great in most cases, I found that being unable to override the default word
> > break rules was quite limiting especially since a lot of these punctuation
> > symbols have important meaning now on the web (@ - mentions, # - hashtags,
> > etc.)
> >
> > I've wrapped this extension to the Standard Tokenizer in an Elasticsearch
> > plugin, which can be found at -
> > [GitHub - bbguitar77/elasticsearch-analysis-standardext](https://github.com/bbguitar77/elasticsearch-analysis-standardext) ...
> > definitely looking for feedback as this is my first go at an Elasticsearch
> > plugin!
> >
> > I'm hoping other Elasticsearch / Lucene users find this helpful.
> >
> > Cheers!
> > Bryan
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/929dc7c3-ff99-43a4-a287-1a8f89d86e3f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/929dc7c3-ff99-43a4-a287-1a8f89d86e3f%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/929dc7c3-ff99-43a4-a287-1a8f89d86e3f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/929dc7c3-ff99-43a4-a287-1a8f89d86e3f%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/CAGdPd5koucm5\_kWjrVqs%3DEQm8fySnuM81i\_j3KGUKi3NafaPpQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGdPd5koucm5_kWjrVqs%3DEQm8fySnuM81i_j3KGUKi3NafaPpQ%40mail.gmail.com)
> [https://groups.google.com/d/msgid/elasticsearch/CAGdPd5koucm5\_kWjrVqs%3DEQm8fySnuM81i\_j3KGUKi3NafaPpQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGdPd5koucm5_kWjrVqs%3DEQm8fySnuM81i_j3KGUKi3NafaPpQ%40mail.gmail.com?utm_medium=email&utm_source=footer)
> .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHDGY%3DV12TgJEmZL9\_snJH7DOpQ2KT6%3DCWqRe8q2sU0GA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHDGY%3DV12TgJEmZL9_snJH7DOpQ2KT6%3DCWqRe8q2sU0GA%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Raj\_Gupta](https://discuss.elastic.co/u/Raj_Gupta)
#### Post date: [September 9, 2014, 12:35pm UTC](https://discuss.elastic.co/t/es-plugin-to-extend-lucenes-standard-tokenizer/19670/4 "2014-09-09T12:35:48Z")
Hi Vineeth,
I haven't looked at the plugin Bryan has created ,
However creating a plugin for special characters gives better performance
over patter tokenizer or custom filters.
Regards,
Raj
On Tuesday, September 9, 2014 9:06:08 AM UTC+5:30, vineeth mohan wrote:
> Hello Bryan ,
>
> Congrats on your first plugin.
> I have a question here - Can you implement the whole plugin by using
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-pattern-tokenizer.html)
> tokenizer ?
>
> Is your plugin providing any advantage over going this approach ?
>
> Thanks
> Vineeth
>
> On Tue, Sep 9, 2014 at 7:56 AM, Bryan Warner \<[bryan....@gmail.com](mailto:bryan....@gmail.com)
> \\> wrote:
>
> > Hi all,
> >
> > Recently, I've been working on an extension to Lucene's Standard
> > Tokenizer that allows the user to customize / override the default word
> > boundary break rules for Unicode characters. The Standard Tokenizer
> > implements the word break rules from the Unicode Text segmentation
> > [http://www.unicode.org/reports/tr29/](http://www.unicode.org/reports/tr29/) algorithm where most punctuation
> > symbols (except for underscore '\_') are treated as hard word breaks (e.g.
> > "@foo" , "#foo" are tokenized to "foo"). While the Standard Tokenizer works
> > great in most cases, I found that being unable to override the default word
> > break rules was quite limiting especially since a lot of these punctuation
> > symbols have important meaning now on the web (@ - mentions, # - hashtags,
> > etc.)
> >
> > I've wrapped this extension to the Standard Tokenizer in an Elasticsearch
> > plugin, which can be found at -
> > [GitHub - bbguitar77/elasticsearch-analysis-standardext](https://github.com/bbguitar77/elasticsearch-analysis-standardext) ...
> > definitely looking for feedback as this is my first go at an Elasticsearch
> > plugin!
> >
> > I'm hoping other Elasticsearch / Lucene users find this helpful.
> >
> > Cheers!
> > Bryan
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/929dc7c3-ff99-43a4-a287-1a8f89d86e3f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/929dc7c3-ff99-43a4-a287-1a8f89d86e3f%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/929dc7c3-ff99-43a4-a287-1a8f89d86e3f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/929dc7c3-ff99-43a4-a287-1a8f89d86e3f%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/dd2fd3b0-f6c1-40e0-b2d7-723084027354%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/dd2fd3b0-f6c1-40e0-b2d7-723084027354%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Bryan\_Warner](https://discuss.elastic.co/u/Bryan_Warner)
#### Post date: [September 9, 2014, 2:37pm UTC](https://discuss.elastic.co/t/es-plugin-to-extend-lucenes-standard-tokenizer/19670/5 "2014-09-09T14:37:21Z")
You can definitely use the Pattern Tokenizer to define your own token
separators (i.e. the word boundary breaks), but you will add complexity and
lose some of the benefits of the StandardTokenizer.
First, regarding complexity, if you want certain characters to not become
token separators, you're regex will start becoming increasingly complex and
lengthy ... I know from experience as I have used the Pattern Tokenizer
previously.
Second, you will lose all the benefits of the StandardTokenizer and how it
handles international characters / symbols and special characters outside
the ASCII range of Unicode characters (e.g. Left-to-right markers, etc.).
It's probably impossible to capture all that the StandardTokenizer does in
a regex expression
If you are dealing with very clean data and perhaps English-only text, then
the Pattern Tokenizer could be a viable solution. But, when dealing with
web / user-generated data and many languages, then the StandardTokenizer is
your friend.
- Bryan
On Monday, September 8, 2014 11:36:08 PM UTC-4, vineeth mohan wrote:
> Hello Bryan ,
>
> Congrats on your first plugin.
> I have a question here - Can you implement the whole plugin by using
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-pattern-tokenizer.html)
> tokenizer ?
>
> Is your plugin providing any advantage over going this approach ?
>
> Thanks
> Vineeth
>
> On Tue, Sep 9, 2014 at 7:56 AM, Bryan Warner \<[bryan....@gmail.com](mailto:bryan....@gmail.com)
> \\> wrote:
>
> > Hi all,
> >
> > Recently, I've been working on an extension to Lucene's Standard
> > Tokenizer that allows the user to customize / override the default word
> > boundary break rules for Unicode characters. The Standard Tokenizer
> > implements the word break rules from the Unicode Text segmentation
> > [http://www.unicode.org/reports/tr29/](http://www.unicode.org/reports/tr29/) algorithm where most punctuation
> > symbols (except for underscore '\_') are treated as hard word breaks (e.g.
> > "@foo" , "#foo" are tokenized to "foo"). While the Standard Tokenizer works
> > great in most cases, I found that being unable to override the default word
> > break rules was quite limiting especially since a lot of these punctuation
> > symbols have important meaning now on the web (@ - mentions, # - hashtags,
> > etc.)
> >
> > I've wrapped this extension to the Standard Tokenizer in an Elasticsearch
> > plugin, which can be found at -
> > [GitHub - bbguitar77/elasticsearch-analysis-standardext](https://github.com/bbguitar77/elasticsearch-analysis-standardext) ...
> > definitely looking for feedback as this is my first go at an Elasticsearch
> > plugin!
> >
> > I'm hoping other Elasticsearch / Lucene users find this helpful.
> >
> > Cheers!
> > Bryan
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/929dc7c3-ff99-43a4-a287-1a8f89d86e3f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/929dc7c3-ff99-43a4-a287-1a8f89d86e3f%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/929dc7c3-ff99-43a4-a287-1a8f89d86e3f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/929dc7c3-ff99-43a4-a287-1a8f89d86e3f%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/06e1a147-e7b3-43b3-ab9a-6e4bbef4a63f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/06e1a147-e7b3-43b3-ab9a-6e4bbef4a63f%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:03am UTC](https://discuss.elastic.co/t/es-plugin-to-extend-lucenes-standard-tokenizer/19670/6 "2017-07-06T01:03:38Z")