# PatternTokenizer?

**URL:** <https://discuss.elastic.co/t/patterntokenizer/3945>\
**Category:** Elasticsearch\
**Created:** [February 18, 2011, 10:47am UTC](https://discuss.elastic.co/t/patterntokenizer/3945 "2011-02-18T10:47:07Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pawel\_Wrzeszcz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pawel_wrzeszcz/32/3267_2.png) [@Pawel\_Wrzeszcz](https://discuss.elastic.co/u/Pawel_Wrzeszcz)\
**Post date:** [February 18, 2011, 10:47am UTC](https://discuss.elastic.co/t/patterntokenizer/3945/1 "2011-02-18T10:47:07Z")

</div>

Hello,

I have encountered the following ElasticSearch "puzzle":

1. I would like my analyzer to use ASCII Folding token filter (to make  
search work properly in Polish).

2. I need fine-grained control for the way tokens are split. Therefore  
I use PatternAnalyzer.

I cannot combine these two, because (correct me if I am wrong) the  
only analyzer that allows customization of filters is CustomAnalyzer  
and I cannot add ASCI Folding filer to PatternAnalyzer.

I guess that having PatternTokenizer (not analyzer) would solve my  
problem. Actually, there is (private) class  
PatternAnalyzer.PatternTokenizer in Lucene.

Would it make sense to add PatternTokenizer to ElasticSearch? Or is  
there any other way to solve issue?

Regards,  
-Pawel Wrzeszcz

---

<div class="post-metadata">

**Author:** ![rmuir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rmuir/32/44949_2.png) [@rmuir](https://discuss.elastic.co/u/rmuir)\
**Post date:** [February 18, 2011, 9:13pm UTC](https://discuss.elastic.co/t/patterntokenizer/3945/2 "2011-02-18T21:13:07Z")

</div>

On Fri, Feb 18, 2011 at 5:47 AM, Pawel Wrzeszcz  
[pawel.wrzeszcz@gmail.com](mailto:pawel.wrzeszcz@gmail.com) wrote:

> I guess that having PatternTokenizer (not analyzer) would solve my  
> problem. Actually, there is (private) class  
> PatternAnalyzer.PatternTokenizer in Lucene.
> 
> Would it make sense to add PatternTokenizer to Elasticsearch? Or is  
> there any other way to solve issue?

Hi, in lucene's trunk there are some cleaner pattern-based components  
that replaced this PatternAnalyzer: a PatternTokenizer,  
PatternTokenFilter, and PatternCharFilter. These used to be in Solr  
but in Lucene's trunk all analysis components are merged into this  
single module.

[http://svn.apache.org/repos/asf/lucene/dev/trunk/modules/analysis/common/src/java/org/apache/lucene/analysis/pattern/](http://svn.apache.org/repos/asf/lucene/dev/trunk/modules/analysis/common/src/java/org/apache/lucene/analysis/pattern/)

(Note, for performance reasons some of these use new features of  
upcoming Lucene 3.1's analysis API, but maybe would still be easier to  
start from)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:11am UTC](https://discuss.elastic.co/t/patterntokenizer/3945/3 "2017-07-06T04:11:51Z")

</div>