# Pattern tokenization

**URL:** <https://discuss.elastic.co/t/pattern-tokenization/4043>\
**Category:** Elasticsearch\
**Created:** [March 5, 2011, 5:42pm UTC](https://discuss.elastic.co/t/pattern-tokenization/4043 "2011-03-05T17:42:40Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![K\_B](https://avatars.discourse-cdn.com/v4/letter/k/278dde/32.png) [@K\_B](https://discuss.elastic.co/u/K_B)\
**Post date:** [March 5, 2011, 5:42pm UTC](https://discuss.elastic.co/t/pattern-tokenization/4043/1 "2011-03-05T17:42:40Z")

</div>

Hello,

I need to tokenize on a special pattern, here is the code I use with  
SOLR:

I tried to implement it in ES using the pattern analyzer, but it didnt  
work out, I tried:  
index :  
analysis :  
analyzer :  
linieAnalzyer :  
type : pattern  
lowercase: false  
pattern: ", "

and realized I need a special tokenizer, so my hope was that during  
analysis the break appears at the apttern level and this would work:

index :  
analysis :  
analyzer :  
linieAnalzyer :  
type : pattern  
lowercase: false  
pattern: ", "  
flags: ""  
tokenizer: keyword

but he ignores my keyword - has anyone an idea how I can tell him to  
treat the string as string, then break it up onto ", " and see the  
resulting bits as keywords that mustnt be treated any further?

Best,

K.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [March 6, 2011, 4:31am UTC](https://discuss.elastic.co/t/pattern-tokenization/4043/2 "2011-03-06T04:31:45Z")

</div>

Thats what the pattern analyzer should do. Its already an fully fledged analyzer, so specifying a tokenizer does not mean much (it only applies to custom analyzers). Can you fist a sample with your config, and using the analyzer API show the output that you get, and where it differs from what you expect?  
On Saturday, March 5, 2011 at 7:42 PM, K.B. wrote:

> Hello,
> 
> I need to tokenize on a special pattern, here is the code I use with  
> SOLR:
> 
> I tried to implement it in ES using the pattern analyzer, but it didnt  
> work out, I tried:  
> index :  
> analysis :  
> analyzer :  
> linieAnalzyer :  
> type : pattern  
> lowercase: false  
> pattern: ", "
> 
> and realized I need a special tokenizer, so my hope was that during  
> analysis the break appears at the apttern level and this would work:
> 
> index :  
> analysis :  
> analyzer :  
> linieAnalzyer :  
> type : pattern  
> lowercase: false  
> pattern: ", "  
> flags: ""  
> tokenizer: keyword
> 
> but he ignores my keyword - has anyone an idea how I can tell him to  
> treat the string as string, then break it up onto ", " and see the  
> resulting bits as keywords that mustnt be treated any further?
> 
> Best,
> 
> K.

---

<div class="post-metadata">

**Author:** ![K\_B](https://avatars.discourse-cdn.com/v4/letter/k/278dde/32.png) [@K\_B](https://discuss.elastic.co/u/K_B)\
**Post date:** [March 6, 2011, 10:36am UTC](https://discuss.elastic.co/t/pattern-tokenization/4043/3 "2011-03-06T10:36:27Z")

</div>

Hi Shay,

thanks for answering. I'll try my best to explain, but don't know how  
exactly to show the output as I'm quite new to ES (coming from SOLR).

My current config now is (in elasticsearch.yml):  
index :  
analysis :  
analyzer :  
linieAnalzyer :  
type : pattern  
lowercase: false  
pattern: ", "  
flags: ""

My config for the type is as follows (from java):

String mapping = "{"produkt": {" +  
""properties" : {" +  
""linie\_mv" : " +  
"{ "type" : "string", " +  
""index" : "analyzed", " +  
" "index\_analyzer": "linieAnalzyer", " +  
" "search\_analyzer": "linieAnalzyer" }" +  
"}}}";

client.admin().indices().create(new  
CreateIndexRequest(index).mapping(type, mapping)).actionGet();

and works like expected;

I then index docs containing in the linie\_mv field values like:

"\_0\_Foo, \_1\_Foo\_Bar, \_2\_Foo\_Bar\_Fii /Faa, \_3\_Foo\_Bar\_Fii /Faa\_La"

its treated at the ", " but then tokenized at all whitespaces " " and  
not as resulting single terms (after breaking it using the ", " he  
mustn't dig in deeper as he then breaks my tree;

Reason is as this is used later as a facet based upon a prefix-query,  
say e.g.:

prefix("_0_")  
facets -\> "\_0\_Foo" ; "\_0\_Foo2" etc.;

prefix"\_1\_Foo"  
-\> facets -\> "\_1\_Foo\_Bar" etc.;

So I can dynamically get the leaves of a tree and the corresponding  
doc's;

Currently I get only part right, most I also get by a prefix("\*") is  
also chunked words from inner like "/Faa\_La" here;

I hope this is now a bit more clear whats going wrong for my case.

Best,

K.

PS: Shay, thanks for your work - as a long-term compass user I really  
appreaciate it very much!

On 6 Mrz., 05:31, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:

> Thats what the pattern analyzer should do. Its already an fully fledged analyzer, so specifying a tokenizer does not mean much (it only applies to custom analyzers). Can you fist a sample with your config, and using the analyzer API show the output that you get, and where it differs from what you expect?
> 
> On Saturday, March 5, 2011 at 7:42 PM, K.B. wrote:
> 
> > Hello,
> 
> > I need to tokenize on a special pattern, here is the code I use with  
> > SOLR:
> 
> > 
> 
> > I tried to implement it in ES using the pattern analyzer, but it didnt  
> > work out, I tried:  
> > index :  
> > analysis :  
> > analyzer :  
> > linieAnalzyer :  
> > type : pattern  
> > lowercase: false  
> > pattern: ", "
> 
> > and realized I need a special tokenizer, so my hope was that during  
> > analysis the break appears at the apttern level and this would work:
> 
> > index :  
> > analysis :  
> > analyzer :  
> > linieAnalzyer :  
> > type : pattern  
> > lowercase: false  
> > pattern: ", "  
> > flags: ""  
> > tokenizer: keyword
> 
> > but he ignores my keyword - has anyone an idea how I can tell him to  
> > treat the string as string, then break it up onto ", " and see the  
> > resulting bits as keywords that mustnt be treated any further?
> 
> > Best,
> 
> > K.

---

<div class="post-metadata">

**Author:** ![K\_B](https://avatars.discourse-cdn.com/v4/letter/k/278dde/32.png) [@K\_B](https://discuss.elastic.co/u/K_B)\
**Post date:** [March 7, 2011, 6:49pm UTC](https://discuss.elastic.co/t/pattern-tokenization/4043/4 "2011-03-07T18:49:24Z")

</div>

Hi Shay,

I'm a bit forther now. Instead of trying to continue arguing with the  
pattern analyzer I instead went to "keyword" and did a  
((String) o[37]).split(", ")  
in java so it now indexes as it expected:

linie\_mv: [  
" \_0\_foo"  
" \_1\_foo\_bar"  
" \_2\_foo\_bar\_xxxx"  
" \_3\_foo\_bar\_yyyy"  
]

When I now query using java-api:

SearchRequestBuilder builder = client.prepareSearch(index);  
XContentQueryBuilder qb = QueryBuilders.queryString("\_1\_foo\*")  
.defaultOperator(QueryStringQueryBuilder.Operator.OR)  
.field("linie\_mv")  
.field("id")  
.allowLeadingWildcard(true).useDisMax(true);

```
    builder.setQuery(qb);

```

builder.addFacet(FacetBuilders.termsFacet("linie\_mv").field("linie\_mv"));

I get the expected resultset. The facet here, however, doesn't return  
the limited facet one expects (" \_1\_foo\_bar") but instead returns  
_all_ the facets that any matching item has. I also tried to use a  
FacetFilter like  
.facetFilter(new TermsFilterBuilder("linie\_mv", "\_1\_foo\*")  
but it has no effect as still all facets are returned....

I managed to bypass this at the moment by introducing special fields  
for each line e.g.: linie\_mv\_0 ... linie\_mv\_5 but this won't work when  
doing m:m tree mappings - any idea what I can do or how I can limit  
the facets to the one containing special characters? (I expected  
facetFilter to just do this?)

Best,

Korbinian

On 6 Mrz., 11:36, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote:

> Hi Shay,
> 
> thanks for answering. I'll try my best to explain, but don't know how  
> exactly to show the output as I'm quite new to ES (coming from SOLR).
> 
> My current config now is (in elasticsearch.yml):  
> index :  
> analysis :  
> analyzer :  
> linieAnalzyer :  
> type : pattern  
> lowercase: false  
> pattern: ", "  
> flags: ""
> 
> My config for the type is as follows (from java):
> 
> String mapping = "{"produkt": {" +  
> ""properties" : {" +  
> ""linie\_mv" : " +  
> "{ "type" : "string", " +  
> ""index" : "analyzed", " +  
> " "index\_analyzer": "linieAnalzyer", " +  
> " "search\_analyzer": "linieAnalzyer" }" +  
> "}}}";
> 
> client.admin().indices().create(new  
> CreateIndexRequest(index).mapping(type, mapping)).actionGet();
> 
> and works like expected;
> 
> I then index docs containing in the linie\_mv field values like:
> 
> "\_0\_Foo, \_1\_Foo\_Bar, \_2\_Foo\_Bar\_Fii /Faa, \_3\_Foo\_Bar\_Fii /Faa\_La"
> 
> its treated at the ", " but then tokenized at all whitespaces " " and  
> not as resulting single terms (after breaking it using the ", " he  
> mustn't dig in deeper as he then breaks my tree;
> 
> Reason is as this is used later as a facet based upon a prefix-query,  
> say e.g.:
> 
> prefix("_0_")  
> facets -\> "\_0\_Foo" ; "\_0\_Foo2" etc.;
> 
> prefix"\_1\_Foo"  
> -\> facets -\> "\_1\_Foo\_Bar" etc.;
> 
> So I can dynamically get the leaves of a tree and the corresponding  
> doc's;
> 
> Currently I get only part right, most I also get by a prefix("\*") is  
> also chunked words from inner like "/Faa\_La" here;
> 
> I hope this is now a bit more clear whats going wrong for my case.
> 
> Best,
> 
> K.
> 
> PS: Shay, thanks for your work - as a long-term compass user I really  
> appreaciate it very much!
> 
> On 6 Mrz., 05:31, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > Thats what the pattern analyzer should do. Its already an fully fledged analyzer, so specifying a tokenizer does not mean much (it only applies to custom analyzers). Can you fist a sample with your config, and using the analyzer API show the output that you get, and where it differs from what you expect?
> 
> > On Saturday, March 5, 2011 at 7:42 PM, K.B. wrote:
> > 
> > > Hello,
> 
> > > I need to tokenize on a special pattern, here is the code I use with  
> > > SOLR:
> 
> > >
> 
> > > I tried to implement it in ES using the pattern analyzer, but it didnt  
> > > work out, I tried:  
> > > index :  
> > > analysis :  
> > > analyzer :  
> > > linieAnalzyer :  
> > > type : pattern  
> > > lowercase: false  
> > > pattern: ", "
> 
> > > and realized I need a special tokenizer, so my hope was that during  
> > > analysis the break appears at the apttern level and this would work:
> 
> > > index :  
> > > analysis :  
> > > analyzer :  
> > > linieAnalzyer :  
> > > type : pattern  
> > > lowercase: false  
> > > pattern: ", "  
> > > flags: ""  
> > > tokenizer: keyword
> 
> > > but he ignores my keyword - has anyone an idea how I can tell him to  
> > > treat the string as string, then break it up onto ", " and see the  
> > > resulting bits as keywords that mustnt be treated any further?
> 
> > > Best,
> 
> > > K.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [March 8, 2011, 6:43am UTC](https://discuss.elastic.co/t/pattern-tokenization/4043/5 "2011-03-08T06:43:23Z")

</div>

Facet (non global ones) are restricted to the query you execute, if that query matches more than you expect, then you need to optimize it. I am not sure if it should or not, since its hard to follow whats going on. In order for me to easily help, just gist a curl recreation and I can have a look ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/help)).  
On Monday, March 7, 2011 at 8:49 PM, K.B. wrote:

> Hi Shay,
> 
> I'm a bit forther now. Instead of trying to continue arguing with the  
> pattern analyzer I instead went to "keyword" and did a  
> ((String) o[37]).split(", ")  
> in java so it now indexes as it expected:
> 
> linie\_mv: [  
> " \_0\_foo"  
> " \_1\_foo\_bar"  
> " \_2\_foo\_bar\_xxxx"  
> " \_3\_foo\_bar\_yyyy"  
> ]
> 
> When I now query using java-api:
> 
> SearchRequestBuilder builder = client.prepareSearch(index);  
> XContentQueryBuilder qb = QueryBuilders.queryString("\_1\_foo\*")  
> .defaultOperator(QueryStringQueryBuilder.Operator.OR)  
> .field("linie\_mv")  
> .field("id")  
> .allowLeadingWildcard(true).useDisMax(true);
> 
> builder.setQuery(qb);
> 
> builder.addFacet(FacetBuilders.termsFacet("linie\_mv").field("linie\_mv"));
> 
> I get the expected resultset. The facet here, however, doesn't return  
> the limited facet one expects (" \_1\_foo\_bar") but instead returns  
> _all_ the facets that any matching item has. I also tried to use a  
> FacetFilter like  
> .facetFilter(new TermsFilterBuilder("linie\_mv", "\_1\_foo\*")  
> but it has no effect as still all facets are returned....
> 
> I managed to bypass this at the moment by introducing special fields  
> for each line e.g.: linie\_mv\_0 ... linie\_mv\_5 but this won't work when  
> doing m:m tree mappings - any idea what I can do or how I can limit  
> the facets to the one containing special characters? (I expected  
> facetFilter to just do this?)
> 
> Best,
> 
> Korbinian
> 
> On 6 Mrz., 11:36, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote:
> 
> > Hi Shay,
> > 
> > thanks for answering. I'll try my best to explain, but don't know how  
> > exactly to show the output as I'm quite new to ES (coming from SOLR).
> > 
> > My current config now is (in elasticsearch.yml):  
> > index :  
> > analysis :  
> > analyzer :  
> > linieAnalzyer :  
> > type : pattern  
> > lowercase: false  
> > pattern: ", "  
> > flags: ""
> > 
> > My config for the type is as follows (from java):
> > 
> > String mapping = "{"produkt": {" +  
> > ""properties" : {" +  
> > ""linie\_mv" : " +  
> > "{ "type" : "string", " +  
> > ""index" : "analyzed", " +  
> > " "index\_analyzer": "linieAnalzyer", " +  
> > " "search\_analyzer": "linieAnalzyer" }" +  
> > "}}}";
> > 
> > client.admin().indices().create(new  
> > CreateIndexRequest(index).mapping(type, mapping)).actionGet();
> > 
> > and works like expected;
> > 
> > I then index docs containing in the linie\_mv field values like:
> > 
> > "\_0\_Foo, \_1\_Foo\_Bar, \_2\_Foo\_Bar\_Fii /Faa, \_3\_Foo\_Bar\_Fii /Faa\_La"
> > 
> > its treated at the ", " but then tokenized at all whitespaces " " and  
> > not as resulting single terms (after breaking it using the ", " he  
> > mustn't dig in deeper as he then breaks my tree;
> > 
> > Reason is as this is used later as a facet based upon a prefix-query,  
> > say e.g.:
> > 
> > prefix("_0_")  
> > facets -\> "\_0\_Foo" ; "\_0\_Foo2" etc.;
> > 
> > prefix"\_1\_Foo"  
> > -\> facets -\> "\_1\_Foo\_Bar" etc.;
> > 
> > So I can dynamically get the leaves of a tree and the corresponding  
> > doc's;
> > 
> > Currently I get only part right, most I also get by a prefix("\*") is  
> > also chunked words from inner like "/Faa\_La" here;
> > 
> > I hope this is now a bit more clear whats going wrong for my case.
> > 
> > Best,
> > 
> > K.
> > 
> > PS: Shay, thanks for your work - as a long-term compass user I really  
> > appreaciate it very much!
> > 
> > On 6 Mrz., 05:31, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > 
> > > Thats what the pattern analyzer should do. Its already an fully fledged analyzer, so specifying a tokenizer does not mean much (it only applies to custom analyzers). Can you fist a sample with your config, and using the analyzer API show the output that you get, and where it differs from what you expect?
> > 
> > > On Saturday, March 5, 2011 at 7:42 PM, K.B. wrote:
> > > 
> > > > Hello,
> > 
> > > > I need to tokenize on a special pattern, here is the code I use with  
> > > > SOLR:
> > 
> > > >
> > 
> > > > I tried to implement it in ES using the pattern analyzer, but it didnt  
> > > > work out, I tried:  
> > > > index :  
> > > > analysis :  
> > > > analyzer :  
> > > > linieAnalzyer :  
> > > > type : pattern  
> > > > lowercase: false  
> > > > pattern: ", "
> > 
> > > > and realized I need a special tokenizer, so my hope was that during  
> > > > analysis the break appears at the apttern level and this would work:
> > 
> > > > index :  
> > > > analysis :  
> > > > analyzer :  
> > > > linieAnalzyer :  
> > > > type : pattern  
> > > > lowercase: false  
> > > > pattern: ", "  
> > > > flags: ""  
> > > > tokenizer: keyword
> > 
> > > > but he ignores my keyword - has anyone an idea how I can tell him to  
> > > > treat the string as string, then break it up onto ", " and see the  
> > > > resulting bits as keywords that mustnt be treated any further?
> > 
> > > > Best,
> > 
> > > > K.

---

<div class="post-metadata">

**Author:** ![K\_B](https://avatars.discourse-cdn.com/v4/letter/k/278dde/32.png) [@K\_B](https://discuss.elastic.co/u/K_B)\
**Post date:** [March 8, 2011, 9:41am UTC](https://discuss.elastic.co/t/pattern-tokenization/4043/6 "2011-03-08T09:41:46Z")

</div>

Hello Shay,

I try to re-explain. Please note that my problem relates to the  
_logic_ behind, not the query itself and as I'm using the java-api  
there typically mustnt be any simple error within the query itself.

Now, re-explained:

Think about a dealer like amazon: you have books that can be mapped to  
different tree-based categories, say

-\> Category  
-\> Themes  
and are sometimes limited to the area where they may be delivered to  
due to legal reasons;

Now imagine you have 3 books:

A: "Historic-\>Old" (category) thats in the themes: "Religion-\>Jews"  
and mustnt be sent to people living in pakistan  
B: "Historic-\>Old" (category) thats in the themes: "Religion-\>Jews"  
and it may be delivered everywhere;  
C: "Historic-\>New" (category) thats in the themes: "Religion-

> Christians" and it mustnt be sent to people living in pakistan

Now, the Categorys as well as the Themes have to be displayed using a  
tree like structure as we have so many more categories and themes; If  
you now are a looking at the restrictions these shouldn't be served to  
our customers from e.g: pakistan, so a pre-created tree won't fit here  
as we only know at runtime after a limiting search what type and kind  
of trees we can show them.

As we don't know how many of those linkings are going to exist at any  
time and we don't want to alter or browse for schemes (which is hard  
if you have no clue how these will be called) we come to the point  
what makes a good search solution vs. a simple, stupid ones like we  
have in databases (and no, token and stemming isn't the thing that  
RDBMs havent....): faceting

If we can now facet over the result of a search (here: only items  
appropriate for our customer) we can dynamically retrieve _all_  
existing tree-nodes that make sense for the user to be presented to.

So to overcome we need to do following:

1. put together all possible mappings at 1(!) place and that we call  
"line" (which means it will be a multi-valued field for lucene)
2. these mappings have to express the whole mapping route as we can't  
query too much around without sacrifying too much performance, meaning  
mappings will be called "Religion", "Religion\_Jews" for our Jews  
category
3. we need to be able to tell the depth of each node/ leaf so we can  
go for it and get all nodes from a certain level, resulting in our new  
mapping like "\_0\_Religion", "\_1\_Religion\_Jews" etc.

This then translates to our following JSON-Books for ES:  
[Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754) (the part under  
JavaScript - my mistake, sorry)

These are then indexed and now we want to query for them from Java,  
and here we want _all_ books for _all_ countries but see the full tree  
then retrieved by a facet for the level 0 (think about a that the sub-  
trees have all 1 common root under 0), so we want to get facets with  
values:

"\_0\_Religion"  
"\_0\_Historic"

so we do in java a query: "Java query level 0" [Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)

but instead of the expecting with facetFilter { new  
TermsFilterBuilder("linie\_mv", "_0_\*") } we get all multi-value-field  
entered values as all books match the query, so the filter for the  
facet didn't work. However, we have another way: regex that might  
help, so we do:  
-\> "Java" [Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)

but then it happens: if we enter regex("_0__") we get back an empty  
facet, meaning =\> TermsFacet: []  
if we enter regex("__0_\*") we get a not nice error from ES: ERROR -\>

> <https://gist.github.com/kbachl/933bc88c63d30e9f2754>
>
> There are more than three files. show original

I hope this makes it more clear what I have to achive and what didn't  
work out as expected. Any help on solving this is really appreciated  
as if I can't fix this somehow very soon I have to ditch ES,

Best,

K.

On 8 Mrz., 07:43, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:

> Facet (non global ones) are restricted to the query you execute, if that query matches more than you expect, then you need to optimize it. I am not sure if it should or not, since its hard to follow whats going on. In order for me to easily help, just gist a curl recreation and I can have a look ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/help)).
> 
> On Monday, March 7, 2011 at 8:49 PM, K.B. wrote:
> 
> > Hi Shay,
> 
> > I'm a bit forther now. Instead of trying to continue arguing with the  
> > pattern analyzer I instead went to "keyword" and did a  
> > ((String) o[37]).split(", ")  
> > in java so it now indexes as it expected:
> 
> > linie\_mv: [  
> > " \_0\_foo"  
> > " \_1\_foo\_bar"  
> > " \_2\_foo\_bar\_xxxx"  
> > " \_3\_foo\_bar\_yyyy"  
> > ]
> 
> > When I now query using java-api:
> 
> > SearchRequestBuilder builder = client.prepareSearch(index);  
> > XContentQueryBuilder qb = QueryBuilders.queryString("\_1\_foo\*")  
> > .defaultOperator(QueryStringQueryBuilder.Operator.OR)  
> > .field("linie\_mv")  
> > .field("id")  
> > .allowLeadingWildcard(true).useDisMax(true);
> 
> > builder.setQuery(qb);
> 
> > builder.addFacet(FacetBuilders.termsFacet("linie\_mv").field("linie\_mv"));
> 
> > I get the expected resultset. The facet here, however, doesn't return  
> > the limited facet one expects (" \_1\_foo\_bar") but instead returns  
> > _all_ the facets that any matching item has. I also tried to use a  
> > FacetFilter like  
> > .facetFilter(new TermsFilterBuilder("linie\_mv", "\_1\_foo\*")  
> > but it has no effect as still all facets are returned....
> 
> > I managed to bypass this at the moment by introducing special fields  
> > for each line e.g.: linie\_mv\_0 ... linie\_mv\_5 but this won't work when  
> > doing m:m tree mappings - any idea what I can do or how I can limit  
> > the facets to the one containing special characters? (I expected  
> > facetFilter to just do this?)
> 
> > Best,
> 
> > Korbinian
> 
> > On 6 Mrz., 11:36, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote:
> > 
> > > Hi Shay,
> 
> > > thanks for answering. I'll try my best to explain, but don't know how  
> > > exactly to show the output as I'm quite new to ES (coming from SOLR).
> 
> > > My current config now is (in elasticsearch.yml):  
> > > index :  
> > > analysis :  
> > > analyzer :  
> > > linieAnalzyer :  
> > > type : pattern  
> > > lowercase: false  
> > > pattern: ", "  
> > > flags: ""
> 
> > > My config for the type is as follows (from java):
> 
> > > String mapping = "{"produkt": {" +  
> > > ""properties" : {" +  
> > > ""linie\_mv" : " +  
> > > "{ "type" : "string", " +  
> > > ""index" : "analyzed", " +  
> > > " "index\_analyzer": "linieAnalzyer", " +  
> > > " "search\_analyzer": "linieAnalzyer" }" +  
> > > "}}}";
> 
> > > client.admin().indices().create(new  
> > > CreateIndexRequest(index).mapping(type, mapping)).actionGet();
> 
> > > and works like expected;
> 
> > > I then index docs containing in the linie\_mv field values like:
> 
> > > "\_0\_Foo, \_1\_Foo\_Bar, \_2\_Foo\_Bar\_Fii /Faa, \_3\_Foo\_Bar\_Fii /Faa\_La"
> 
> > > its treated at the ", " but then tokenized at all whitespaces " " and  
> > > not as resulting single terms (after breaking it using the ", " he  
> > > mustn't dig in deeper as he then breaks my tree;
> 
> > > Reason is as this is used later as a facet based upon a prefix-query,  
> > > say e.g.:
> 
> > > prefix("_0_")  
> > > facets -\> "\_0\_Foo" ; "\_0\_Foo2" etc.;
> 
> > > prefix"\_1\_Foo"  
> > > -\> facets -\> "\_1\_Foo\_Bar" etc.;
> 
> > > So I can dynamically get the leaves of a tree and the corresponding  
> > > doc's;
> 
> > > Currently I get only part right, most I also get by a prefix("\*") is  
> > > also chunked words from inner like "/Faa\_La" here;
> 
> > > I hope this is now a bit more clear whats going wrong for my case.
> 
> > > Best,
> 
> > > K.
> 
> > > PS: Shay, thanks for your work - as a long-term compass user I really  
> > > appreaciate it very much!
> 
> > > On 6 Mrz., 05:31, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > > > Thats what the pattern analyzer should do. Its already an fully fledged analyzer, so specifying a tokenizer does not mean much (it only applies to custom analyzers). Can you fist a sample with your config, and using the analyzer API show the output that you get, and where it differs from what you expect?
> 
> > > > On Saturday, March 5, 2011 at 7:42 PM, K.B. wrote:
> > > > 
> > > > > Hello,
> 
> > > > > I need to tokenize on a special pattern, here is the code I use with  
> > > > > SOLR:
> 
> > > > >
> 
> > > > > I tried to implement it in ES using the pattern analyzer, but it didnt  
> > > > > work out, I tried:  
> > > > > index :  
> > > > > analysis :  
> > > > > analyzer :  
> > > > > linieAnalzyer :  
> > > > > type : pattern  
> > > > > lowercase: false  
> > > > > pattern: ", "
> 
> > > > > and realized I need a special tokenizer, so my hope was that during  
> > > > > analysis the break appears at the apttern level and this would work:
> 
> > > > > index :  
> > > > > analysis :  
> > > > > analyzer :  
> > > > > linieAnalzyer :  
> > > > > type : pattern  
> > > > > lowercase: false  
> > > > > pattern: ", "  
> > > > > flags: ""  
> > > > > tokenizer: keyword
> 
> > > > > but he ignores my keyword - has anyone an idea how I can tell him to  
> > > > > treat the string as string, then break it up onto ", " and see the  
> > > > > resulting bits as keywords that mustnt be treated any further?
> 
> > > > > Best,
> 
> > > > > K.

---

<div class="post-metadata">

**Author:** ![K\_B](https://avatars.discourse-cdn.com/v4/letter/k/278dde/32.png) [@K\_B](https://discuss.elastic.co/u/K_B)\
**Post date:** [March 8, 2011, 9:46am UTC](https://discuss.elastic.co/t/pattern-tokenization/4043/7 "2011-03-08T09:46:54Z")

</div>

PS: just to note: in this system one has to query for each level from  
0 to n-1 where n is the node level he wants to reach; as n is limited  
to a reasonable low value this is no problem as our problem will still  
be solved in complexity O(n) and not the feared O(n^kn) (explained:  
simple complexity and not an exponential increase)

On 8 Mrz., 10:41, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote:

> Hello Shay,
> 
> I try to re-explain. Please note that my problem relates to the  
> _logic_ behind, not the query itself and as I'm using the java-api  
> there typically mustnt be any simple error within the query itself.
> 
> Now, re-explained:
> 
> Think about a dealer like amazon: you have books that can be mapped to  
> different tree-based categories, say
> 
> -\> Category  
> -\> Themes  
> and are sometimes limited to the area where they may be delivered to  
> due to legal reasons;
> 
> Now imagine you have 3 books:
> 
> A: "Historic-\>Old" (category) thats in the themes: "Religion-\>Jews"  
> and mustnt be sent to people living in pakistan  
> B: "Historic-\>Old" (category) thats in the themes: "Religion-\>Jews"  
> and it may be delivered everywhere;  
> C: "Historic-\>New" (category) thats in the themes: "Religion-
> 
> > Christians" and it mustnt be sent to people living in pakistan
> 
> Now, the Categorys as well as the Themes have to be displayed using a  
> tree like structure as we have so many more categories and themes; If  
> you now are a looking at the restrictions these shouldn't be served to  
> our customers from e.g: pakistan, so a pre-created tree won't fit here  
> as we only know at runtime after a limiting search what type and kind  
> of trees we can show them.
> 
> As we don't know how many of those linkings are going to exist at any  
> time and we don't want to alter or browse for schemes (which is hard  
> if you have no clue how these will be called) we come to the point  
> what makes a good search solution vs. a simple, stupid ones like we  
> have in databases (and no, token and stemming isn't the thing that  
> RDBMs havent....): faceting
> 
> If we can now facet over the result of a search (here: only items  
> appropriate for our customer) we can dynamically retrieve _all_  
> existing tree-nodes that make sense for the user to be presented to.
> 
> So to overcome we need to do following:
> 
> 1. put together all possible mappings at 1(!) place and that we call  
> "line" (which means it will be a multi-valued field for lucene)
> 2. these mappings have to express the whole mapping route as we can't  
> query too much around without sacrifying too much performance, meaning  
> mappings will be called "Religion", "Religion\_Jews" for our Jews  
> category
> 3. we need to be able to tell the depth of each node/ leaf so we can  
> go for it and get all nodes from a certain level, resulting in our new  
> mapping like "\_0\_Religion", "\_1\_Religion\_Jews" etc.
> 
> This then translates to our following JSON-Books for ES:[Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)(the part under  
> JavaScript - my mistake, sorry)
> 
> These are then indexed and now we want to query for them from Java,  
> and here we want _all_ books for _all_ countries but see the full tree  
> then retrieved by a facet for the level 0 (think about a that the sub-  
> trees have all 1 common root under 0), so we want to get facets with  
> values:
> 
> "\_0\_Religion"  
> "\_0\_Historic"
> 
> so we do in java a query: "Java query level 0" [Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)
> 
> but instead of the expecting with facetFilter { new  
> TermsFilterBuilder("linie\_mv", "_0_\*") } we get all multi-value-field  
> entered values as all books match the query, so the filter for the  
> facet didn't work. However, we have another way: regex that might  
> help, so we do:  
> -\> "Java"[Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)
> 
> but then it happens: if we enter regex("_0__") we get back an empty  
> facet, meaning =\> TermsFacet: []  
> if we enter regex("__0_\*") we get a not nice error from ES: ERROR -\>[Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)
> 
> I hope this makes it more clear what I have to achive and what didn't  
> work out as expected. Any help on solving this is really appreciated  
> as if I can't fix this somehow very soon I have to ditch ES,
> 
> Best,
> 
> K.
> 
> On 8 Mrz., 07:43, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > Facet (non global ones) are restricted to the query you execute, if that query matches more than you expect, then you need to optimize it. I am not sure if it should or not, since its hard to follow whats going on. In order for me to easily help, just gist a curl recreation and I can have a look ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/help)).
> 
> > On Monday, March 7, 2011 at 8:49 PM, K.B. wrote:
> > 
> > > Hi Shay,
> 
> > > I'm a bit forther now. Instead of trying to continue arguing with the  
> > > pattern analyzer I instead went to "keyword" and did a  
> > > ((String) o[37]).split(", ")  
> > > in java so it now indexes as it expected:
> 
> > > linie\_mv: [  
> > > " \_0\_foo"  
> > > " \_1\_foo\_bar"  
> > > " \_2\_foo\_bar\_xxxx"  
> > > " \_3\_foo\_bar\_yyyy"  
> > > ]
> 
> > > When I now query using java-api:
> 
> > > SearchRequestBuilder builder = client.prepareSearch(index);  
> > > XContentQueryBuilder qb = QueryBuilders.queryString("\_1\_foo\*")  
> > > .defaultOperator(QueryStringQueryBuilder.Operator.OR)  
> > > .field("linie\_mv")  
> > > .field("id")  
> > > .allowLeadingWildcard(true).useDisMax(true);
> 
> > > builder.setQuery(qb);
> 
> > > builder.addFacet(FacetBuilders.termsFacet("linie\_mv").field("linie\_mv"));
> 
> > > I get the expected resultset. The facet here, however, doesn't return  
> > > the limited facet one expects (" \_1\_foo\_bar") but instead returns  
> > > _all_ the facets that any matching item has. I also tried to use a  
> > > FacetFilter like  
> > > .facetFilter(new TermsFilterBuilder("linie\_mv", "\_1\_foo\*")  
> > > but it has no effect as still all facets are returned....
> 
> > > I managed to bypass this at the moment by introducing special fields  
> > > for each line e.g.: linie\_mv\_0 ... linie\_mv\_5 but this won't work when  
> > > doing m:m tree mappings - any idea what I can do or how I can limit  
> > > the facets to the one containing special characters? (I expected  
> > > facetFilter to just do this?)
> 
> > > Best,
> 
> > > Korbinian
> 
> > > On 6 Mrz., 11:36, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote:
> > > 
> > > > Hi Shay,
> 
> > > > thanks for answering. I'll try my best to explain, but don't know how  
> > > > exactly to show the output as I'm quite new to ES (coming from SOLR).
> 
> > > > My current config now is (in elasticsearch.yml):  
> > > > index :  
> > > > analysis :  
> > > > analyzer :  
> > > > linieAnalzyer :  
> > > > type : pattern  
> > > > lowercase: false  
> > > > pattern: ", "  
> > > > flags: ""
> 
> > > > My config for the type is as follows (from java):
> 
> > > > String mapping = "{"produkt": {" +  
> > > > ""properties" : {" +  
> > > > ""linie\_mv" : " +  
> > > > "{ "type" : "string", " +  
> > > > ""index" : "analyzed", " +  
> > > > " "index\_analyzer": "linieAnalzyer", " +  
> > > > " "search\_analyzer": "linieAnalzyer" }" +  
> > > > "}}}";
> 
> > > > client.admin().indices().create(new  
> > > > CreateIndexRequest(index).mapping(type, mapping)).actionGet();
> 
> > > > and works like expected;
> 
> > > > I then index docs containing in the linie\_mv field values like:
> 
> > > > "\_0\_Foo, \_1\_Foo\_Bar, \_2\_Foo\_Bar\_Fii /Faa, \_3\_Foo\_Bar\_Fii /Faa\_La"
> 
> > > > its treated at the ", " but then tokenized at all whitespaces " " and  
> > > > not as resulting single terms (after breaking it using the ", " he  
> > > > mustn't dig in deeper as he then breaks my tree;
> 
> > > > Reason is as this is used later as a facet based upon a prefix-query,  
> > > > say e.g.:
> 
> > > > prefix("_0_")  
> > > > facets -\> "\_0\_Foo" ; "\_0\_Foo2" etc.;
> 
> > > > prefix"\_1\_Foo"  
> > > > -\> facets -\> "\_1\_Foo\_Bar" etc.;
> 
> > > > So I can dynamically get the leaves of a tree and the corresponding  
> > > > doc's;
> 
> > > > Currently I get only part right, most I also get by a prefix("\*") is  
> > > > also chunked words from inner like "/Faa\_La" here;
> 
> > > > I hope this is now a bit more clear whats going wrong for my case.
> 
> > > > Best,
> 
> > > > K.
> 
> > > > PS: Shay, thanks for your work - as a long-term compass user I really  
> > > > appreaciate it very much!
> 
> > > > On 6 Mrz., 05:31, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > > > > Thats what the pattern analyzer should do. Its already an fully fledged analyzer, so specifying a tokenizer does not mean much (it only applies to custom analyzers). Can you fist a sample with your config, and using the analyzer API show the output that you get, and where it differs from what you expect?
> 
> > > > > On Saturday, March 5, 2011 at 7:42 PM, K.B. wrote:
> > > > > 
> > > > > > Hello,
> 
> > > > > > I need to tokenize on a special pattern, here is the code I use with  
> > > > > > SOLR:
> 
> > > > > >
> 
> > > > > > I tried to implement it in ES using the pattern analyzer, but it didnt  
> > > > > > work out, I tried:  
> > > > > > index :  
> > > > > > analysis :  
> > > > > > analyzer :  
> > > > > > linieAnalzyer :  
> > > > > > type : pattern  
> > > > > > lowercase: false  
> > > > > > pattern: ", "
> 
> > > > > > and realized I need a special tokenizer, so my hope was that during  
> > > > > > analysis the break appears at the apttern level and this would work:
> 
> > > > > > index :  
> > > > > > analysis :  
> > > > > > analyzer :  
> > > > > > linieAnalzyer :  
> > > > > > type : pattern  
> > > > > > lowercase: false  
> > > > > > pattern: ", "  
> > > > > > flags: ""  
> > > > > > tokenizer: keyword
> 
> > > > > > but he ignores my keyword - has anyone an idea how I can tell him to  
> > > > > > treat the string as string, then break it up onto ", " and see the  
> > > > > > resulting bits as keywords that mustnt be treated any further?
> 
> > > > > > Best,
> 
> > > > > > K.

---

<div class="post-metadata">

**Author:** ![K\_B](https://avatars.discourse-cdn.com/v4/letter/k/278dde/32.png) [@K\_B](https://discuss.elastic.co/u/K_B)\
**Post date:** [March 8, 2011, 7:40pm UTC](https://discuss.elastic.co/t/pattern-tokenization/4043/8 "2011-03-08T19:40:02Z")

</div>

Ok, problem solved thanks to Lukáš Vlček!

Problem is/ was that the regex-origin was not made clear in doc,  
instead of:  
.regex("_1__")  
one has to use  
.regex("^1._")  
as this is not lucene or JavaScript expression but a regular JAVA  
regular expression!

PS: Wish for future to all coming features and additions to ES: please  
add more verbose doc with more telling examples and the origin of  
expressions!

On 8 Mrz., 10:46, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote:

> PS: just to note: in this system one has to query for each level from  
> 0 to n-1 where n is the node level he wants to reach; as n is limited  
> to a reasonable low value this is no problem as our problem will still  
> be solved in complexity O(n) and not the feared O(n^kn) (explained:  
> simple complexity and not an exponential increase)
> 
> On 8 Mrz., 10:41, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote:
> 
> > Hello Shay,
> 
> > I try to re-explain. Please note that my problem relates to the  
> > _logic_ behind, not the query itself and as I'm using the java-api  
> > there typically mustnt be any simple error within the query itself.
> 
> > Now, re-explained:
> 
> > Think about a dealer like amazon: you have books that can be mapped to  
> > different tree-based categories, say
> 
> > -\> Category  
> > -\> Themes  
> > and are sometimes limited to the area where they may be delivered to  
> > due to legal reasons;
> 
> > Now imagine you have 3 books:
> 
> > A: "Historic-\>Old" (category) thats in the themes: "Religion-\>Jews"  
> > and mustnt be sent to people living in pakistan  
> > B: "Historic-\>Old" (category) thats in the themes: "Religion-\>Jews"  
> > and it may be delivered everywhere;  
> > C: "Historic-\>New" (category) thats in the themes: "Religion-
> 
> > > Christians" and it mustnt be sent to people living in pakistan
> 
> > Now, the Categorys as well as the Themes have to be displayed using a  
> > tree like structure as we have so many more categories and themes; If  
> > you now are a looking at the restrictions these shouldn't be served to  
> > our customers from e.g: pakistan, so a pre-created tree won't fit here  
> > as we only know at runtime after a limiting search what type and kind  
> > of trees we can show them.
> 
> > As we don't know how many of those linkings are going to exist at any  
> > time and we don't want to alter or browse for schemes (which is hard  
> > if you have no clue how these will be called) we come to the point  
> > what makes a good search solution vs. a simple, stupid ones like we  
> > have in databases (and no, token and stemming isn't the thing that  
> > RDBMs havent....): faceting
> 
> > If we can now facet over the result of a search (here: only items  
> > appropriate for our customer) we can dynamically retrieve _all_  
> > existing tree-nodes that make sense for the user to be presented to.
> 
> > So to overcome we need to do following:
> 
> > 1. put together all possible mappings at 1(!) place and that we call  
> > "line" (which means it will be a multi-valued field for lucene)
> > 2. these mappings have to express the whole mapping route as we can't  
> > query too much around without sacrifying too much performance, meaning  
> > mappings will be called "Religion", "Religion\_Jews" for our Jews  
> > category
> > 3. we need to be able to tell the depth of each node/ leaf so we can  
> > go for it and get all nodes from a certain level, resulting in our new  
> > mapping like "\_0\_Religion", "\_1\_Religion\_Jews" etc.
> 
> > This then translates to our following JSON-Books for ES:[Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)(thepart under  
> > JavaScript - my mistake, sorry)
> 
> > These are then indexed and now we want to query for them from Java,  
> > and here we want _all_ books for _all_ countries but see the full tree  
> > then retrieved by a facet for the level 0 (think about a that the sub-  
> > trees have all 1 common root under 0), so we want to get facets with  
> > values:
> 
> > "\_0\_Religion"  
> > "\_0\_Historic"
> 
> > so we do in java a query: "Java query level 0" [Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)
> 
> > but instead of the expecting with facetFilter { new  
> > TermsFilterBuilder("linie\_mv", "_0_\*") } we get all multi-value-field  
> > entered values as all books match the query, so the filter for the  
> > facet didn't work. However, we have another way: regex that might  
> > help, so we do:  
> > -\> "Java"[Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)
> 
> > but then it happens: if we enter regex("_0__") we get back an empty  
> > facet, meaning =\> TermsFacet: []  
> > if we enter regex("__0_\*") we get a not nice error from ES: ERROR -\>[Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)
> 
> > I hope this makes it more clear what I have to achive and what didn't  
> > work out as expected. Any help on solving this is really appreciated  
> > as if I can't fix this somehow very soon I have to ditch ES,
> 
> > Best,
> 
> > K.
> 
> > On 8 Mrz., 07:43, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > > Facet (non global ones) are restricted to the query you execute, if that query matches more than you expect, then you need to optimize it. I am not sure if it should or not, since its hard to follow whats going on. In order for me to easily help, just gist a curl recreation and I can have a look ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/help)).
> 
> > > On Monday, March 7, 2011 at 8:49 PM, K.B. wrote:
> > > 
> > > > Hi Shay,
> 
> > > > I'm a bit forther now. Instead of trying to continue arguing with the  
> > > > pattern analyzer I instead went to "keyword" and did a  
> > > > ((String) o[37]).split(", ")  
> > > > in java so it now indexes as it expected:
> 
> > > > linie\_mv: [  
> > > > " \_0\_foo"  
> > > > " \_1\_foo\_bar"  
> > > > " \_2\_foo\_bar\_xxxx"  
> > > > " \_3\_foo\_bar\_yyyy"  
> > > > ]
> 
> > > > When I now query using java-api:
> 
> > > > SearchRequestBuilder builder = client.prepareSearch(index);  
> > > > XContentQueryBuilder qb = QueryBuilders.queryString("\_1\_foo\*")  
> > > > .defaultOperator(QueryStringQueryBuilder.Operator.OR)  
> > > > .field("linie\_mv")  
> > > > .field("id")  
> > > > .allowLeadingWildcard(true).useDisMax(true);
> 
> > > > builder.setQuery(qb);
> 
> > > > builder.addFacet(FacetBuilders.termsFacet("linie\_mv").field("linie\_mv"));
> 
> > > > I get the expected resultset. The facet here, however, doesn't return  
> > > > the limited facet one expects (" \_1\_foo\_bar") but instead returns  
> > > > _all_ the facets that any matching item has. I also tried to use a  
> > > > FacetFilter like  
> > > > .facetFilter(new TermsFilterBuilder("linie\_mv", "\_1\_foo\*")  
> > > > but it has no effect as still all facets are returned....
> 
> > > > I managed to bypass this at the moment by introducing special fields  
> > > > for each line e.g.: linie\_mv\_0 ... linie\_mv\_5 but this won't work when  
> > > > doing m:m tree mappings - any idea what I can do or how I can limit  
> > > > the facets to the one containing special characters? (I expected  
> > > > facetFilter to just do this?)
> 
> > > > Best,
> 
> > > > Korbinian
> 
> > > > On 6 Mrz., 11:36, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote:
> > > > 
> > > > > Hi Shay,
> 
> > > > > thanks for answering. I'll try my best to explain, but don't know how  
> > > > > exactly to show the output as I'm quite new to ES (coming from SOLR).
> 
> > > > > My current config now is (in elasticsearch.yml):  
> > > > > index :  
> > > > > analysis :  
> > > > > analyzer :  
> > > > > linieAnalzyer :  
> > > > > type : pattern  
> > > > > lowercase: false  
> > > > > pattern: ", "  
> > > > > flags: ""
> 
> > > > > My config for the type is as follows (from java):
> 
> > > > > String mapping = "{"produkt": {" +  
> > > > > ""properties" : {" +  
> > > > > ""linie\_mv" : " +  
> > > > > "{ "type" : "string", " +  
> > > > > ""index" : "analyzed", " +  
> > > > > " "index\_analyzer": "linieAnalzyer", " +  
> > > > > " "search\_analyzer": "linieAnalzyer" }" +  
> > > > > "}}}";
> 
> > > > > client.admin().indices().create(new  
> > > > > CreateIndexRequest(index).mapping(type, mapping)).actionGet();
> 
> > > > > and works like expected;
> 
> > > > > I then index docs containing in the linie\_mv field values like:
> 
> > > > > "\_0\_Foo, \_1\_Foo\_Bar, \_2\_Foo\_Bar\_Fii /Faa, \_3\_Foo\_Bar\_Fii /Faa\_La"
> 
> > > > > its treated at the ", " but then tokenized at all whitespaces " " and  
> > > > > not as resulting single terms (after breaking it using the ", " he  
> > > > > mustn't dig in deeper as he then breaks my tree;
> 
> > > > > Reason is as this is used later as a facet based upon a prefix-query,  
> > > > > say e.g.:
> 
> > > > > prefix("_0_")  
> > > > > facets -\> "\_0\_Foo" ; "\_0\_Foo2" etc.;
> 
> > > > > prefix"\_1\_Foo"  
> > > > > -\> facets -\> "\_1\_Foo\_Bar" etc.;
> 
> > > > > So I can dynamically get the leaves of a tree and the corresponding  
> > > > > doc's;
> 
> > > > > Currently I get only part right, most I also get by a prefix("\*") is  
> > > > > also chunked words from inner like "/Faa\_La" here;
> 
> > > > > I hope this is now a bit more clear whats going wrong for my case.
> 
> > > > > Best,
> 
> > > > > K.
> 
> > > > > PS: Shay, thanks for your work - as a long-term compass user I really  
> > > > > appreaciate it very much!
> 
> > > > > On 6 Mrz., 05:31, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > > > > > Thats what the pattern analyzer should do. Its already an fully fledged analyzer, so specifying a tokenizer does not mean much (it only applies to custom analyzers). Can you fist a sample with your config, and using the analyzer API show the output that you get, and where it differs from what you expect?
> 
> > > > > > On Saturday, March 5, 2011 at 7:42 PM, K.B. wrote:
> > > > > > 
> > > > > > > Hello,
> 
> > > > > > > I need to tokenize on a special pattern, here is the code I use with  
> > > > > > > SOLR:
> 
> > > > > > >
> 
> > > > > > > I tried to implement it in ES using the pattern analyzer, but it didnt  
> > > > > > > work out, I tried:  
> > > > > > > index :  
> > > > > > > analysis :  
> > > > > > > analyzer :  
> > > > > > > linieAnalzyer :  
> > > > > > > type : pattern  
> > > > > > > lowercase: false  
> > > > > > > pattern: ", "
> 
> > > > > > > and realized I need a special tokenizer, so my hope
> 
> ...
> 
> Erfahren Sie mehr »

---

<div class="post-metadata">

**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\
**Post date:** [March 9, 2011, 8:50am UTC](https://discuss.elastic.co/t/pattern-tokenization/4043/9 "2011-03-09T08:50:28Z")

</div>

Hi,

I added link to Java Pattern API to ES docs. Also feel free to report other  
missing references or you can fork the docs and send pull request 🙂

Regards,  
Lukas

On Tue, Mar 8, 2011 at 8:40 PM, K.B. [korbinian.bachl@googlemail.com](mailto:korbinian.bachl@googlemail.com) wrote:

> Ok, problem solved thanks to Lukáš Vlček!
> 
> Problem is/ was that the regex-origin was not made clear in doc,  
> instead of:  
> .regex("_1__")  
> one has to use  
> .regex("^1._")  
> as this is not lucene or JavaScript expression but a regular JAVA  
> regular expression!
> 
> PS: Wish for future to all coming features and additions to ES: please  
> add more verbose doc with more telling examples and the origin of  
> expressions!
> 
> On 8 Mrz., 10:46, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote:
> 
> > PS: just to note: in this system one has to query for each level from  
> > 0 to n-1 where n is the node level he wants to reach; as n is limited  
> > to a reasonable low value this is no problem as our problem will still  
> > be solved in complexity O(n) and not the feared O(n^kn) (explained:  
> > simple complexity and not an exponential increase)
> > 
> > On 8 Mrz., 10:41, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote:
> > 
> > > Hello Shay,
> > 
> > > I try to re-explain. Please note that my problem relates to the  
> > > _logic_ behind, not the query itself and as I'm using the java-api  
> > > there typically mustnt be any simple error within the query itself.
> > 
> > > Now, re-explained:
> > 
> > > Think about a dealer like amazon: you have books that can be mapped to  
> > > different tree-based categories, say
> > 
> > > -\> Category  
> > > -\> Themes  
> > > and are sometimes limited to the area where they may be delivered to  
> > > due to legal reasons;
> > 
> > > Now imagine you have 3 books:
> > 
> > > A: "Historic-\>Old" (category) thats in the themes: "Religion-\>Jews"  
> > > and mustnt be sent to people living in pakistan  
> > > B: "Historic-\>Old" (category) thats in the themes: "Religion-\>Jews"  
> > > and it may be delivered everywhere;  
> > > C: "Historic-\>New" (category) thats in the themes: "Religion-
> > 
> > > > Christians" and it mustnt be sent to people living in pakistan
> > 
> > > Now, the Categorys as well as the Themes have to be displayed using a  
> > > tree like structure as we have so many more categories and themes; If  
> > > you now are a looking at the restrictions these shouldn't be served to  
> > > our customers from e.g: pakistan, so a pre-created tree won't fit here  
> > > as we only know at runtime after a limiting search what type and kind  
> > > of trees we can show them.
> > 
> > > As we don't know how many of those linkings are going to exist at any  
> > > time and we don't want to alter or browse for schemes (which is hard  
> > > if you have no clue how these will be called) we come to the point  
> > > what makes a good search solution vs. a simple, stupid ones like we  
> > > have in databases (and no, token and stemming isn't the thing that  
> > > RDBMs havent....): faceting
> > 
> > > If we can now facet over the result of a search (here: only items  
> > > appropriate for our customer) we can dynamically retrieve _all_  
> > > existing tree-nodes that make sense for the user to be presented to.
> > 
> > > So to overcome we need to do following:
> > 
> > > 1. put together all possible mappings at 1(!) place and that we call  
> > > "line" (which means it will be a multi-valued field for lucene)
> > > 2. these mappings have to express the whole mapping route as we can't  
> > > query too much around without sacrifying too much performance, meaning  
> > > mappings will be called "Religion", "Religion\_Jews" for our Jews  
> > > category
> > > 3. we need to be able to tell the depth of each node/ leaf so we can  
> > > go for it and get all nodes from a certain level, resulting in our new  
> > > mapping like "\_0\_Religion", "\_1\_Religion\_Jews" etc.
> > 
> > > This then translates to our following JSON-Books for ES:  
> > > [Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)(thepart under  
> > > JavaScript - my mistake, sorry)
> > 
> > > These are then indexed and now we want to query for them from Java,  
> > > and here we want _all_ books for _all_ countries but see the full tree  
> > > then retrieved by a facet for the level 0 (think about a that the sub-  
> > > trees have all 1 common root under 0), so we want to get facets with  
> > > values:
> > 
> > > "\_0\_Religion"  
> > > "\_0\_Historic"
> > 
> > > so we do in java a query: "Java query level 0"  
> > > [Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)
> > 
> > > but instead of the expecting with facetFilter { new  
> > > TermsFilterBuilder("linie\_mv", "_0_\*") } we get all multi-value-field  
> > > entered values as all books match the query, so the filter for the  
> > > facet didn't work. However, we have another way: regex that might  
> > > help, so we do:  
> > > -\> "Java"[Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)
> > 
> > > but then it happens: if we enter regex("_0__") we get back an empty  
> > > facet, meaning =\> TermsFacet: []  
> > > if we enter regex("__0_\*") we get a not nice error from ES: ERROR -\>  
> > > [Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)
> > 
> > > I hope this makes it more clear what I have to achive and what didn't  
> > > work out as expected. Any help on solving this is really appreciated  
> > > as if I can't fix this somehow very soon I have to ditch ES,
> > 
> > > Best,
> > 
> > > K.
> > 
> > > On 8 Mrz., 07:43, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > 
> > > > Facet (non global ones) are restricted to the query you execute, if  
> > > > that query matches more than you expect, then you need to optimize it. I am  
> > > > not sure if it should or not, since its hard to follow whats going on. In  
> > > > order for me to easily help, just gist a curl recreation and I can have a  
> > > > look ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/help)).
> > 
> > > > On Monday, March 7, 2011 at 8:49 PM, K.B. wrote:
> > > > 
> > > > > Hi Shay,
> > 
> > > > > I'm a bit forther now. Instead of trying to continue arguing with  
> > > > > the  
> > > > > pattern analyzer I instead went to "keyword" and did a  
> > > > > ((String) o[37]).split(", ")  
> > > > > in java so it now indexes as it expected:
> > 
> > > > > linie\_mv: [  
> > > > > " \_0\_foo"  
> > > > > " \_1\_foo\_bar"  
> > > > > " \_2\_foo\_bar\_xxxx"  
> > > > > " \_3\_foo\_bar\_yyyy"  
> > > > > ]
> > 
> > > > > When I now query using java-api:
> > 
> > > > > SearchRequestBuilder builder = client.prepareSearch(index);  
> > > > > XContentQueryBuilder qb = QueryBuilders.queryString("\_1\_foo\*")  
> > > > > .defaultOperator(QueryStringQueryBuilder.Operator.OR)  
> > > > > .field("linie\_mv")  
> > > > > .field("id")  
> > > > > .allowLeadingWildcard(true).useDisMax(true);
> > 
> > > > > builder.setQuery(qb);
> > 
> > > > >
> 
> builder.addFacet(FacetBuilders.termsFacet("linie\_mv").field("linie\_mv"));
> 
> > > > > I get the expected resultset. The facet here, however, doesn't  
> > > > > return  
> > > > > the limited facet one expects (" \_1\_foo\_bar") but instead returns  
> > > > > _all_ the facets that any matching item has. I also tried to use a  
> > > > > FacetFilter like  
> > > > > .facetFilter(new TermsFilterBuilder("linie\_mv", "\_1\_foo\*")  
> > > > > but it has no effect as still all facets are returned....
> > 
> > > > > I managed to bypass this at the moment by introducing special  
> > > > > fields  
> > > > > for each line e.g.: linie\_mv\_0 ... linie\_mv\_5 but this won't work  
> > > > > when  
> > > > > doing m:m tree mappings - any idea what I can do or how I can limit  
> > > > > the facets to the one containing special characters? (I expected  
> > > > > facetFilter to just do this?)
> > 
> > > > > Best,
> > 
> > > > > Korbinian
> > 
> > > > > On 6 Mrz., 11:36, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote:
> > > > > 
> > > > > > Hi Shay,
> > 
> > > > > > thanks for answering. I'll try my best to explain, but don't know  
> > > > > > how  
> > > > > > exactly to show the output as I'm quite new to ES (coming from  
> > > > > > SOLR).
> > 
> > > > > > My current config now is (in elasticsearch.yml):  
> > > > > > index :  
> > > > > > analysis :  
> > > > > > analyzer :  
> > > > > > linieAnalzyer :  
> > > > > > type : pattern  
> > > > > > lowercase: false  
> > > > > > pattern: ", "  
> > > > > > flags: ""
> > 
> > > > > > My config for the type is as follows (from java):
> > 
> > > > > > String mapping = "{"produkt": {" +  
> > > > > > ""properties" : {" +  
> > > > > > ""linie\_mv" : " +  
> > > > > > "{ "type" : "string", " +  
> > > > > > ""index" : "analyzed", " +  
> > > > > > " "index\_analyzer": "linieAnalzyer", " +  
> > > > > > " "search\_analyzer": "linieAnalzyer" }" +  
> > > > > > "}}}";
> > 
> > > > > > client.admin().indices().create(new  
> > > > > > CreateIndexRequest(index).mapping(type, mapping)).actionGet();
> > 
> > > > > > and works like expected;
> > 
> > > > > > I then index docs containing in the linie\_mv field values like:
> > 
> > > > > > "\_0\_Foo, \_1\_Foo\_Bar, \_2\_Foo\_Bar\_Fii /Faa, \_3\_Foo\_Bar\_Fii /Faa\_La"
> > 
> > > > > > its treated at the ", " but then tokenized at all whitespaces " "  
> > > > > > and  
> > > > > > not as resulting single terms (after breaking it using the ", "  
> > > > > > he  
> > > > > > mustn't dig in deeper as he then breaks my tree;
> > 
> > > > > > Reason is as this is used later as a facet based upon a  
> > > > > > prefix-query,  
> > > > > > say e.g.:
> > 
> > > > > > prefix("_0_")  
> > > > > > facets -\> "\_0\_Foo" ; "\_0\_Foo2" etc.;
> > 
> > > > > > prefix"\_1\_Foo"  
> > > > > > -\> facets -\> "\_1\_Foo\_Bar" etc.;
> > 
> > > > > > So I can dynamically get the leaves of a tree and the  
> > > > > > corresponding  
> > > > > > doc's;
> > 
> > > > > > Currently I get only part right, most I also get by a prefix("\*")  
> > > > > > is  
> > > > > > also chunked words from inner like "/Faa\_La" here;
> > 
> > > > > > I hope this is now a bit more clear whats going wrong for my  
> > > > > > case.
> > 
> > > > > > Best,
> > 
> > > > > > K.
> > 
> > > > > > PS: Shay, thanks for your work - as a long-term compass user I  
> > > > > > really  
> > > > > > appreaciate it very much!
> > 
> > > > > > On 6 Mrz., 05:31, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com)  
> > > > > > wrote:
> > 
> > > > > > > Thats what the pattern analyzer should do. Its already an fully  
> > > > > > > fledged analyzer, so specifying a tokenizer does not mean much (it only  
> > > > > > > applies to custom analyzers). Can you fist a sample with your config, and  
> > > > > > > using the analyzer API show the output that you get, and where it differs  
> > > > > > > from what you expect?
> > 
> > > > > > > On Saturday, March 5, 2011 at 7:42 PM, K.B. wrote:
> > > > > > > 
> > > > > > > > Hello,
> > 
> > > > > > > > I need to tokenize on a special pattern, here is the code I  
> > > > > > > > use with  
> > > > > > > > SOLR:
> > 
> > > > > > > >
> > 
> > > > > > > > I tried to implement it in ES using the pattern analyzer, but  
> > > > > > > > it didnt  
> > > > > > > > work out, I tried:  
> > > > > > > > index :  
> > > > > > > > analysis :  
> > > > > > > > analyzer :  
> > > > > > > > linieAnalzyer :  
> > > > > > > > type : pattern  
> > > > > > > > lowercase: false  
> > > > > > > > pattern: ", "
> > 
> > > > > > > > and realized I need a special tokenizer, so my hope
> > 
> > ...
> > 
> > Erfahren Sie mehr »

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [March 9, 2011, 9:18am UTC](https://discuss.elastic.co/t/pattern-tokenization/4043/10 "2011-03-09T09:18:28Z")

</div>

Right, its a Java regex, I will update the docs. What is a Lucene regex, don't think there is one. All this thread would have been much simpler if you could have just provided what was asked for, a simple curl recreation using the analyze API to see what is the result of the analyzer you created.  
On Tuesday, March 8, 2011 at 9:40 PM, K.B. wrote:

> Ok, problem solved thanks to Lukáš Vlček!
> 
> Problem is/ was that the regex-origin was not made clear in doc,  
> instead of:  
> .regex("_1__")  
> one has to use  
> .regex("^1._")  
> as this is not lucene or JavaScript expression but a regular JAVA  
> regular expression!
> 
> PS: Wish for future to all coming features and additions to ES: please  
> add more verbose doc with more telling examples and the origin of  
> expressions!
> 
> On 8 Mrz., 10:46, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote:
> 
> > PS: just to note: in this system one has to query for each level from  
> > 0 to n-1 where n is the node level he wants to reach; as n is limited  
> > to a reasonable low value this is no problem as our problem will still  
> > be solved in complexity O(n) and not the feared O(n^kn) (explained:  
> > simple complexity and not an exponential increase)
> > 
> > On 8 Mrz., 10:41, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote:
> > 
> > > Hello Shay,
> > 
> > > I try to re-explain. Please note that my problem relates to the  
> > > _logic_ behind, not the query itself and as I'm using the java-api  
> > > there typically mustnt be any simple error within the query itself.
> > 
> > > Now, re-explained:
> > 
> > > Think about a dealer like amazon: you have books that can be mapped to  
> > > different tree-based categories, say
> > 
> > > -\> Category  
> > > -\> Themes  
> > > and are sometimes limited to the area where they may be delivered to  
> > > due to legal reasons;
> > 
> > > Now imagine you have 3 books:
> > 
> > > A: "Historic-\>Old" (category) thats in the themes: "Religion-\>Jews"  
> > > and mustnt be sent to people living in pakistan  
> > > B: "Historic-\>Old" (category) thats in the themes: "Religion-\>Jews"  
> > > and it may be delivered everywhere;  
> > > C: "Historic-\>New" (category) thats in the themes: "Religion-
> > 
> > > > Christians" and it mustnt be sent to people living in pakistan
> > 
> > > Now, the Categorys as well as the Themes have to be displayed using a  
> > > tree like structure as we have so many more categories and themes; If  
> > > you now are a looking at the restrictions these shouldn't be served to  
> > > our customers from e.g: pakistan, so a pre-created tree won't fit here  
> > > as we only know at runtime after a limiting search what type and kind  
> > > of trees we can show them.
> > 
> > > As we don't know how many of those linkings are going to exist at any  
> > > time and we don't want to alter or browse for schemes (which is hard  
> > > if you have no clue how these will be called) we come to the point  
> > > what makes a good search solution vs. a simple, stupid ones like we  
> > > have in databases (and no, token and stemming isn't the thing that  
> > > RDBMs havent....): faceting
> > 
> > > If we can now facet over the result of a search (here: only items  
> > > appropriate for our customer) we can dynamically retrieve _all_  
> > > existing tree-nodes that make sense for the user to be presented to.
> > 
> > > So to overcome we need to do following:
> > 
> > > 1. put together all possible mappings at 1(!) place and that we call  
> > > "line" (which means it will be a multi-valued field for lucene)
> > > 2. these mappings have to express the whole mapping route as we can't  
> > > query too much around without sacrifying too much performance, meaning  
> > > mappings will be called "Religion", "Religion\_Jews" for our Jews  
> > > category
> > > 3. we need to be able to tell the depth of each node/ leaf so we can  
> > > go for it and get all nodes from a certain level, resulting in our new  
> > > mapping like "\_0\_Religion", "\_1\_Religion\_Jews" etc.
> > 
> > > This then translates to our following JSON-Books for ES:[Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)(thepart under  
> > > JavaScript - my mistake, sorry)
> > 
> > > These are then indexed and now we want to query for them from Java,  
> > > and here we want _all_ books for _all_ countries but see the full tree  
> > > then retrieved by a facet for the level 0 (think about a that the sub-  
> > > trees have all 1 common root under 0), so we want to get facets with  
> > > values:
> > 
> > > "\_0\_Religion"  
> > > "\_0\_Historic"
> > 
> > > so we do in java a query: "Java query level 0" [Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)
> > 
> > > but instead of the expecting with facetFilter { new  
> > > TermsFilterBuilder("linie\_mv", "_0_\*") } we get all multi-value-field  
> > > entered values as all books match the query, so the filter for the  
> > > facet didn't work. However, we have another way: regex that might  
> > > help, so we do:  
> > > -\> "Java"[Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)
> > 
> > > but then it happens: if we enter regex("_0__") we get back an empty  
> > > facet, meaning =\> TermsFacet: []  
> > > if we enter regex("__0_\*") we get a not nice error from ES: ERROR -\>[Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)
> > 
> > > I hope this makes it more clear what I have to achive and what didn't  
> > > work out as expected. Any help on solving this is really appreciated  
> > > as if I can't fix this somehow very soon I have to ditch ES,
> > 
> > > Best,
> > 
> > > K.
> > 
> > > On 8 Mrz., 07:43, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > 
> > > > Facet (non global ones) are restricted to the query you execute, if that query matches more than you expect, then you need to optimize it. I am not sure if it should or not, since its hard to follow whats going on. In order for me to easily help, just gist a curl recreation and I can have a look ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/help)).
> > 
> > > > On Monday, March 7, 2011 at 8:49 PM, K.B. wrote:
> > > > 
> > > > > Hi Shay,
> > 
> > > > > I'm a bit forther now. Instead of trying to continue arguing with the  
> > > > > pattern analyzer I instead went to "keyword" and did a  
> > > > > ((String) o[37]).split(", ")  
> > > > > in java so it now indexes as it expected:
> > 
> > > > > linie\_mv: [  
> > > > > " \_0\_foo"  
> > > > > " \_1\_foo\_bar"  
> > > > > " \_2\_foo\_bar\_xxxx"  
> > > > > " \_3\_foo\_bar\_yyyy"  
> > > > > ]
> > 
> > > > > When I now query using java-api:
> > 
> > > > > SearchRequestBuilder builder = client.prepareSearch(index);  
> > > > > XContentQueryBuilder qb = QueryBuilders.queryString("\_1\_foo\*")  
> > > > > .defaultOperator(QueryStringQueryBuilder.Operator.OR)  
> > > > > .field("linie\_mv")  
> > > > > .field("id")  
> > > > > .allowLeadingWildcard(true).useDisMax(true);
> > 
> > > > > builder.setQuery(qb);
> > 
> > > > > builder.addFacet(FacetBuilders.termsFacet("linie\_mv").field("linie\_mv"));
> > 
> > > > > I get the expected resultset. The facet here, however, doesn't return  
> > > > > the limited facet one expects (" \_1\_foo\_bar") but instead returns  
> > > > > _all_ the facets that any matching item has. I also tried to use a  
> > > > > FacetFilter like  
> > > > > .facetFilter(new TermsFilterBuilder("linie\_mv", "\_1\_foo\*")  
> > > > > but it has no effect as still all facets are returned....
> > 
> > > > > I managed to bypass this at the moment by introducing special fields  
> > > > > for each line e.g.: linie\_mv\_0 ... linie\_mv\_5 but this won't work when  
> > > > > doing m:m tree mappings - any idea what I can do or how I can limit  
> > > > > the facets to the one containing special characters? (I expected  
> > > > > facetFilter to just do this?)
> > 
> > > > > Best,
> > 
> > > > > Korbinian
> > 
> > > > > On 6 Mrz., 11:36, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote:
> > > > > 
> > > > > > Hi Shay,
> > 
> > > > > > thanks for answering. I'll try my best to explain, but don't know how  
> > > > > > exactly to show the output as I'm quite new to ES (coming from SOLR).
> > 
> > > > > > My current config now is (in elasticsearch.yml):  
> > > > > > index :  
> > > > > > analysis :  
> > > > > > analyzer :  
> > > > > > linieAnalzyer :  
> > > > > > type : pattern  
> > > > > > lowercase: false  
> > > > > > pattern: ", "  
> > > > > > flags: ""
> > 
> > > > > > My config for the type is as follows (from java):
> > 
> > > > > > String mapping = "{"produkt": {" +  
> > > > > > ""properties" : {" +  
> > > > > > ""linie\_mv" : " +  
> > > > > > "{ "type" : "string", " +  
> > > > > > ""index" : "analyzed", " +  
> > > > > > " "index\_analyzer": "linieAnalzyer", " +  
> > > > > > " "search\_analyzer": "linieAnalzyer" }" +  
> > > > > > "}}}";
> > 
> > > > > > client.admin().indices().create(new  
> > > > > > CreateIndexRequest(index).mapping(type, mapping)).actionGet();
> > 
> > > > > > and works like expected;
> > 
> > > > > > I then index docs containing in the linie\_mv field values like:
> > 
> > > > > > "\_0\_Foo, \_1\_Foo\_Bar, \_2\_Foo\_Bar\_Fii /Faa, \_3\_Foo\_Bar\_Fii /Faa\_La"
> > 
> > > > > > its treated at the ", " but then tokenized at all whitespaces " " and  
> > > > > > not as resulting single terms (after breaking it using the ", " he  
> > > > > > mustn't dig in deeper as he then breaks my tree;
> > 
> > > > > > Reason is as this is used later as a facet based upon a prefix-query,  
> > > > > > say e.g.:
> > 
> > > > > > prefix("_0_")  
> > > > > > facets -\> "\_0\_Foo" ; "\_0\_Foo2" etc.;
> > 
> > > > > > prefix"\_1\_Foo"  
> > > > > > -\> facets -\> "\_1\_Foo\_Bar" etc.;
> > 
> > > > > > So I can dynamically get the leaves of a tree and the corresponding  
> > > > > > doc's;
> > 
> > > > > > Currently I get only part right, most I also get by a prefix("\*") is  
> > > > > > also chunked words from inner like "/Faa\_La" here;
> > 
> > > > > > I hope this is now a bit more clear whats going wrong for my case.
> > 
> > > > > > Best,
> > 
> > > > > > K.
> > 
> > > > > > PS: Shay, thanks for your work - as a long-term compass user I really  
> > > > > > appreaciate it very much!
> > 
> > > > > > On 6 Mrz., 05:31, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > 
> > > > > > > Thats what the pattern analyzer should do. Its already an fully fledged analyzer, so specifying a tokenizer does not mean much (it only applies to custom analyzers). Can you fist a sample with your config, and using the analyzer API show the output that you get, and where it differs from what you expect?
> > 
> > > > > > > On Saturday, March 5, 2011 at 7:42 PM, K.B. wrote:
> > > > > > > 
> > > > > > > > Hello,
> > 
> > > > > > > > I need to tokenize on a special pattern, here is the code I use with  
> > > > > > > > SOLR:
> > 
> > > > > > > >
> > 
> > > > > > > > I tried to implement it in ES using the pattern analyzer, but it didnt  
> > > > > > > > work out, I tried:  
> > > > > > > > index :  
> > > > > > > > analysis :  
> > > > > > > > analyzer :  
> > > > > > > > linieAnalzyer :  
> > > > > > > > type : pattern  
> > > > > > > > lowercase: false  
> > > > > > > > pattern: ", "
> > 
> > > > > > > > and realized I need a special tokenizer, so my hope
> > 
> > ...
> > 
> > Erfahren Sie mehr »

---

<div class="post-metadata">

**Author:** ![K\_B](https://avatars.discourse-cdn.com/v4/letter/k/278dde/32.png) [@K\_B](https://discuss.elastic.co/u/K_B)\
**Post date:** [March 9, 2011, 10:03am UTC](https://discuss.elastic.co/t/pattern-tokenization/4043/11 "2011-03-09T10:03:29Z")

</div>

Heya Shay,

yes, next time I will try to be more specific, however this is  
sometimes not quite easy when one is new to a piece of software.  
Especially when it comes to curl its still hard for me as I just don't  
use it (tm) but instead solve all over the java-API from within an  
EJB.

Regarding docs: I already think about forking and doing some additions  
from the knowledge I'm getting with my current project especially some  
kind of "How to treat a danymic Tree - Guide" and "EJB + ES:  
Implementation Suggestions"

Regarding ES from Java: would you suggest to better get 1 Client  
within a Singleton and share this among other requests or should each  
request get hold of his own client?

Best,

K.

On 9 Mrz., 10:18, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:

> Right, its a Java regex, I will update the docs. What is a Lucene regex, don't think there is one. All this thread would have been much simpler if you could have just provided what was asked for, a simple curl recreation using the analyze API to see what is the result of the analyzer you created.
> 
> On Tuesday, March 8, 2011 at 9:40 PM, K.B. wrote:
> 
> > Ok, problem solved thanks to Lukáš Vlček!
> 
> > Problem is/ was that the regex-origin was not made clear in doc,  
> > instead of:  
> > .regex("_1__")  
> > one has to use  
> > .regex("^1._")  
> > as this is not lucene or JavaScript expression but a regular JAVA  
> > regular expression!
> 
> > PS: Wish for future to all coming features and additions to ES: please  
> > add more verbose doc with more telling examples and the origin of  
> > expressions!
> 
> > On 8 Mrz., 10:46, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote:
> > 
> > > PS: just to note: in this system one has to query for each level from  
> > > 0 to n-1 where n is the node level he wants to reach; as n is limited  
> > > to a reasonable low value this is no problem as our problem will still  
> > > be solved in complexity O(n) and not the feared O(n^kn) (explained:  
> > > simple complexity and not an exponential increase)
> 
> > > On 8 Mrz., 10:41, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote:
> 
> > > > Hello Shay,
> 
> > > > I try to re-explain. Please note that my problem relates to the  
> > > > _logic_ behind, not the query itself and as I'm using the java-api  
> > > > there typically mustnt be any simple error within the query itself.
> 
> > > > Now, re-explained:
> 
> > > > Think about a dealer like amazon: you have books that can be mapped to  
> > > > different tree-based categories, say
> 
> > > > -\> Category  
> > > > -\> Themes  
> > > > and are sometimes limited to the area where they may be delivered to  
> > > > due to legal reasons;
> 
> > > > Now imagine you have 3 books:
> 
> > > > A: "Historic-\>Old" (category) thats in the themes: "Religion-\>Jews"  
> > > > and mustnt be sent to people living in pakistan  
> > > > B: "Historic-\>Old" (category) thats in the themes: "Religion-\>Jews"  
> > > > and it may be delivered everywhere;  
> > > > C: "Historic-\>New" (category) thats in the themes: "Religion-
> 
> > > > > Christians" and it mustnt be sent to people living in pakistan
> 
> > > > Now, the Categorys as well as the Themes have to be displayed using a  
> > > > tree like structure as we have so many more categories and themes; If  
> > > > you now are a looking at the restrictions these shouldn't be served to  
> > > > our customers from e.g: pakistan, so a pre-created tree won't fit here  
> > > > as we only know at runtime after a limiting search what type and kind  
> > > > of trees we can show them.
> 
> > > > As we don't know how many of those linkings are going to exist at any  
> > > > time and we don't want to alter or browse for schemes (which is hard  
> > > > if you have no clue how these will be called) we come to the point  
> > > > what makes a good search solution vs. a simple, stupid ones like we  
> > > > have in databases (and no, token and stemming isn't the thing that  
> > > > RDBMs havent....): faceting
> 
> > > > If we can now facet over the result of a search (here: only items  
> > > > appropriate for our customer) we can dynamically retrieve _all_  
> > > > existing tree-nodes that make sense for the user to be presented to.
> 
> > > > So to overcome we need to do following:
> 
> > > > 1. put together all possible mappings at 1(!) place and that we call  
> > > > "line" (which means it will be a multi-valued field for lucene)
> > > > 2. these mappings have to express the whole mapping route as we can't  
> > > > query too much around without sacrifying too much performance, meaning  
> > > > mappings will be called "Religion", "Religion\_Jews" for our Jews  
> > > > category
> > > > 3. we need to be able to tell the depth of each node/ leaf so we can  
> > > > go for it and get all nodes from a certain level, resulting in our new  
> > > > mapping like "\_0\_Religion", "\_1\_Religion\_Jews" etc.
> 
> > > > This then translates to our following JSON-Books for ES:[Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)(thepartunder  
> > > > JavaScript - my mistake, sorry)
> 
> > > > These are then indexed and now we want to query for them from Java,  
> > > > and here we want _all_ books for _all_ countries but see the full tree  
> > > > then retrieved by a facet for the level 0 (think about a that the sub-  
> > > > trees have all 1 common root under 0), so we want to get facets with  
> > > > values:
> 
> > > > "\_0\_Religion"  
> > > > "\_0\_Historic"
> 
> > > > so we do in java a query: "Java query level 0"[Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)
> 
> > > > but instead of the expecting with facetFilter { new  
> > > > TermsFilterBuilder("linie\_mv", "_0_\*") } we get all multi-value-field  
> > > > entered values as all books match the query, so the filter for the  
> > > > facet didn't work. However, we have another way: regex that might  
> > > > help, so we do:  
> > > > -\> "Java"[Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)
> 
> > > > but then it happens: if we enter regex("_0__") we get back an empty  
> > > > facet, meaning =\> TermsFacet: []  
> > > > if we enter regex("__0_\*") we get a not nice error from ES: ERROR -\>[Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)
> 
> > > > I hope this makes it more clear what I have to achive and what didn't  
> > > > work out as expected. Any help on solving this is really appreciated  
> > > > as if I can't fix this somehow very soon I have to ditch ES,
> 
> > > > Best,
> 
> > > > K.
> 
> > > > On 8 Mrz., 07:43, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > > > > Facet (non global ones) are restricted to the query you execute, if that query matches more than you expect, then you need to optimize it. I am not sure if it should or not, since its hard to follow whats going on. In order for me to easily help, just gist a curl recreation and I can have a look ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/help)).
> 
> > > > > On Monday, March 7, 2011 at 8:49 PM, K.B. wrote:
> > > > > 
> > > > > > Hi Shay,
> 
> > > > > > I'm a bit forther now. Instead of trying to continue arguing with the  
> > > > > > pattern analyzer I instead went to "keyword" and did a  
> > > > > > ((String) o[37]).split(", ")  
> > > > > > in java so it now indexes as it expected:
> 
> > > > > > linie\_mv: [  
> > > > > > " \_0\_foo"  
> > > > > > " \_1\_foo\_bar"  
> > > > > > " \_2\_foo\_bar\_xxxx"  
> > > > > > " \_3\_foo\_bar\_yyyy"  
> > > > > > ]
> 
> > > > > > When I now query using java-api:
> 
> > > > > > SearchRequestBuilder builder = client.prepareSearch(index);  
> > > > > > XContentQueryBuilder qb = QueryBuilders.queryString("\_1\_foo\*")  
> > > > > > .defaultOperator(QueryStringQueryBuilder.Operator.OR)  
> > > > > > .field("linie\_mv")  
> > > > > > .field("id")  
> > > > > > .allowLeadingWildcard(true).useDisMax(true);
> 
> > > > > > builder.setQuery(qb);
> 
> > > > > > builder.addFacet(FacetBuilders.termsFacet("linie\_mv").field("linie\_mv"));
> 
> > > > > > I get the expected resultset. The facet here, however, doesn't return  
> > > > > > the limited facet one expects (" \_1\_foo\_bar") but instead returns  
> > > > > > _all_ the facets that any matching item has. I also tried to use a  
> > > > > > FacetFilter like  
> > > > > > .facetFilter(new TermsFilterBuilder("linie\_mv", "\_1\_foo\*")  
> > > > > > but it has no effect as still all facets are returned....
> 
> > > > > > I managed to bypass this at the moment by introducing special fields  
> > > > > > for each line e.g.: linie\_mv\_0 ... linie\_mv\_5 but this won't work when  
> > > > > > doing m:m tree mappings - any idea what I can do or how I can limit  
> > > > > > the facets to the one containing special characters? (I expected  
> > > > > > facetFilter to just do this?)
> 
> > > > > > Best,
> 
> > > > > > Korbinian
> 
> > > > > > On 6 Mrz., 11:36, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote:
> > > > > > 
> > > > > > > Hi Shay,
> 
> > > > > > > thanks for answering. I'll try my best to explain, but don't know how  
> > > > > > > exactly to show the output as I'm quite new to ES (coming from SOLR).
> 
> > > > > > > My current config now is (in elasticsearch.yml):  
> > > > > > > index :  
> > > > > > > analysis :  
> > > > > > > analyzer :  
> > > > > > > linieAnalzyer :  
> > > > > > > type : pattern  
> > > > > > > lowercase: false  
> > > > > > > pattern: ", "  
> > > > > > > flags: ""
> 
> > > > > > > My config for the type is as follows (from java):
> 
> > > > > > > String mapping = "{"produkt": {" +  
> > > > > > > ""properties" : {" +  
> > > > > > > ""linie\_mv" : " +  
> > > > > > > "{ "type" : "string", " +  
> > > > > > > ""index" : "analyzed", " +  
> > > > > > > " "index\_analyzer": "linieAnalzyer", " +  
> > > > > > > " "search\_analyzer": "linieAnalzyer" }" +  
> > > > > > > "}}}";
> 
> > > > > > > client.admin().indices().create(new  
> > > > > > > CreateIndexRequest(index).mapping(type, mapping)).actionGet();
> 
> > > > > > > and works like expected;
> 
> > > > > > > I then index docs containing in the linie\_mv field values like:
> 
> > > > > > > "\_0\_Foo, \_1\_Foo\_Bar, \_2\_Foo\_Bar\_Fii /Faa, \_3\_Foo\_Bar\_Fii /Faa\_La"
> 
> > > > > > > its treated at the ", " but then tokenized at all whitespaces " " and  
> > > > > > > not as resulting single terms (after breaking it using the ", " he  
> > > > > > > mustn't dig in deeper as he then breaks my tree;
> 
> > > > > > > Reason is as this is used later as a facet based upon a prefix-query,  
> > > > > > > say e.g.:
> 
> > > > > > > prefix("_0_")  
> > > > > > > facets -\> "\_0\_Foo" ; "\_0\_Foo2" etc.;
> 
> > > > > > > prefix"\_1\_Foo"  
> > > > > > > -\> facets -\> "\_1\_Foo\_Bar" etc.;
> 
> > > > > > > So I can dynamically get the leaves of a tree and the corresponding  
> > > > > > > doc's;
> 
> > > > > > > Currently I get only part right, most I also get by a prefix("\*") is  
> > > > > > > also chunked words from inner like "/Faa\_La" here;
> 
> > > > > > > I hope this is now a bit more clear whats going wrong for my case.
> 
> > > > > > > Best,
> 
> > > > > > > K.
> 
> > > > > > > PS: Shay, thanks for your work - as a long-term compass user I really  
> > > > > > > appreaciate it very much!
> 
> > > > > > > On 6 Mrz., 05:31, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> ...
> 
> Erfahren Sie mehr »

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [March 9, 2011, 10:05am UTC](https://discuss.elastic.co/t/pattern-tokenization/4043/12 "2011-03-09T10:05:08Z")

</div>

On Wednesday, March 9, 2011 at 12:03 PM, K.B. wrote:  
Heya Shay,

> yes, next time I will try to be more specific, however this is  
> sometimes not quite easy when one is new to a piece of software.  
> Especially when it comes to curl its still hard for me as I just don't  
> use it (tm) but instead solve all over the java-API from within an  
> EJB.
> 
> Regarding docs: I already think about forking and doing some additions  
> from the knowledge I'm getting with my current project especially some  
> kind of "How to treat a danymic Tree - Guide" and "EJB + ES:  
> Implementation Suggestions"  
> Would very much welcome it.
> 
> Regarding ES from Java: would you suggest to better get 1 Client  
> within a Singleton and share this among other requests or should each  
> request get hold of his own client?  
> Single Client, shared among all "requests".
> 
> Best,
> 
> K.
> 
> On 9 Mrz., 10:18, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > Right, its a Java regex, I will update the docs. What is a Lucene regex, don't think there is one. All this thread would have been much simpler if you could have just provided what was asked for, a simple curl recreation using the analyze API to see what is the result of the analyzer you created.
> > 
> > On Tuesday, March 8, 2011 at 9:40 PM, K.B. wrote:
> > 
> > > Ok, problem solved thanks to Lukáš Vlček!
> > 
> > > Problem is/ was that the regex-origin was not made clear in doc,  
> > > instead of:  
> > > .regex("_1__")  
> > > one has to use  
> > > .regex("^1._")  
> > > as this is not lucene or JavaScript expression but a regular JAVA  
> > > regular expression!
> > 
> > > PS: Wish for future to all coming features and additions to ES: please  
> > > add more verbose doc with more telling examples and the origin of  
> > > expressions!
> > 
> > > On 8 Mrz., 10:46, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote:
> > > 
> > > > PS: just to note: in this system one has to query for each level from  
> > > > 0 to n-1 where n is the node level he wants to reach; as n is limited  
> > > > to a reasonable low value this is no problem as our problem will still  
> > > > be solved in complexity O(n) and not the feared O(n^kn) (explained:  
> > > > simple complexity and not an exponential increase)
> > 
> > > > On 8 Mrz., 10:41, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote:
> > 
> > > > > Hello Shay,
> > 
> > > > > I try to re-explain. Please note that my problem relates to the  
> > > > > _logic_ behind, not the query itself and as I'm using the java-api  
> > > > > there typically mustnt be any simple error within the query itself.
> > 
> > > > > Now, re-explained:
> > 
> > > > > Think about a dealer like amazon: you have books that can be mapped to  
> > > > > different tree-based categories, say
> > 
> > > > > -\> Category  
> > > > > -\> Themes  
> > > > > and are sometimes limited to the area where they may be delivered to  
> > > > > due to legal reasons;
> > 
> > > > > Now imagine you have 3 books:
> > 
> > > > > A: "Historic-\>Old" (category) thats in the themes: "Religion-\>Jews"  
> > > > > and mustnt be sent to people living in pakistan  
> > > > > B: "Historic-\>Old" (category) thats in the themes: "Religion-\>Jews"  
> > > > > and it may be delivered everywhere;  
> > > > > C: "Historic-\>New" (category) thats in the themes: "Religion-
> > 
> > > > > > Christians" and it mustnt be sent to people living in pakistan
> > 
> > > > > Now, the Categorys as well as the Themes have to be displayed using a  
> > > > > tree like structure as we have so many more categories and themes; If  
> > > > > you now are a looking at the restrictions these shouldn't be served to  
> > > > > our customers from e.g: pakistan, so a pre-created tree won't fit here  
> > > > > as we only know at runtime after a limiting search what type and kind  
> > > > > of trees we can show them.
> > 
> > > > > As we don't know how many of those linkings are going to exist at any  
> > > > > time and we don't want to alter or browse for schemes (which is hard  
> > > > > if you have no clue how these will be called) we come to the point  
> > > > > what makes a good search solution vs. a simple, stupid ones like we  
> > > > > have in databases (and no, token and stemming isn't the thing that  
> > > > > RDBMs havent....): faceting
> > 
> > > > > If we can now facet over the result of a search (here: only items  
> > > > > appropriate for our customer) we can dynamically retrieve _all_  
> > > > > existing tree-nodes that make sense for the user to be presented to.
> > 
> > > > > So to overcome we need to do following:
> > 
> > > > > 1. put together all possible mappings at 1(!) place and that we call  
> > > > > "line" (which means it will be a multi-valued field for lucene)
> > > > > 2. these mappings have to express the whole mapping route as we can't  
> > > > > query too much around without sacrifying too much performance, meaning  
> > > > > mappings will be called "Religion", "Religion\_Jews" for our Jews  
> > > > > category
> > > > > 3. we need to be able to tell the depth of each node/ leaf so we can  
> > > > > go for it and get all nodes from a certain level, resulting in our new  
> > > > > mapping like "\_0\_Religion", "\_1\_Religion\_Jews" etc.
> > 
> > > > > This then translates to our following JSON-Books for ES:[Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)(thepartunder  
> > > > > JavaScript - my mistake, sorry)
> > 
> > > > > These are then indexed and now we want to query for them from Java,  
> > > > > and here we want _all_ books for _all_ countries but see the full tree  
> > > > > then retrieved by a facet for the level 0 (think about a that the sub-  
> > > > > trees have all 1 common root under 0), so we want to get facets with  
> > > > > values:
> > 
> > > > > "\_0\_Religion"  
> > > > > "\_0\_Historic"
> > 
> > > > > so we do in java a query: "Java query level 0"[Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)
> > 
> > > > > but instead of the expecting with facetFilter { new  
> > > > > TermsFilterBuilder("linie\_mv", "_0_\*") } we get all multi-value-field  
> > > > > entered values as all books match the query, so the filter for the  
> > > > > facet didn't work. However, we have another way: regex that might  
> > > > > help, so we do:  
> > > > > -\> "Java"[Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)
> > 
> > > > > but then it happens: if we enter regex("_0__") we get back an empty  
> > > > > facet, meaning =\> TermsFacet: []  
> > > > > if we enter regex("__0_\*") we get a not nice error from ES: ERROR -\>[Our books: · GitHub](https://gist.github.com/933bc88c63d30e9f2754)
> > 
> > > > > I hope this makes it more clear what I have to achive and what didn't  
> > > > > work out as expected. Any help on solving this is really appreciated  
> > > > > as if I can't fix this somehow very soon I have to ditch ES,
> > 
> > > > > Best,
> > 
> > > > > K.
> > 
> > > > > On 8 Mrz., 07:43, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > 
> > > > > > Facet (non global ones) are restricted to the query you execute, if that query matches more than you expect, then you need to optimize it. I am not sure if it should or not, since its hard to follow whats going on. In order for me to easily help, just gist a curl recreation and I can have a look ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/help)).
> > 
> > > > > > On Monday, March 7, 2011 at 8:49 PM, K.B. wrote:
> > > > > > 
> > > > > > > Hi Shay,
> > 
> > > > > > > I'm a bit forther now. Instead of trying to continue arguing with the  
> > > > > > > pattern analyzer I instead went to "keyword" and did a  
> > > > > > > ((String) o[37]).split(", ")  
> > > > > > > in java so it now indexes as it expected:
> > 
> > > > > > > linie\_mv: [  
> > > > > > > " \_0\_foo"  
> > > > > > > " \_1\_foo\_bar"  
> > > > > > > " \_2\_foo\_bar\_xxxx"  
> > > > > > > " \_3\_foo\_bar\_yyyy"  
> > > > > > > ]
> > 
> > > > > > > When I now query using java-api:
> > 
> > > > > > > SearchRequestBuilder builder = client.prepareSearch(index);  
> > > > > > > XContentQueryBuilder qb = QueryBuilders.queryString("\_1\_foo\*")  
> > > > > > > .defaultOperator(QueryStringQueryBuilder.Operator.OR)  
> > > > > > > .field("linie\_mv")  
> > > > > > > .field("id")  
> > > > > > > .allowLeadingWildcard(true).useDisMax(true);
> > 
> > > > > > > builder.setQuery(qb);
> > 
> > > > > > > builder.addFacet(FacetBuilders.termsFacet("linie\_mv").field("linie\_mv"));
> > 
> > > > > > > I get the expected resultset. The facet here, however, doesn't return  
> > > > > > > the limited facet one expects (" \_1\_foo\_bar") but instead returns  
> > > > > > > _all_ the facets that any matching item has. I also tried to use a  
> > > > > > > FacetFilter like  
> > > > > > > .facetFilter(new TermsFilterBuilder("linie\_mv", "\_1\_foo\*")  
> > > > > > > but it has no effect as still all facets are returned....
> > 
> > > > > > > I managed to bypass this at the moment by introducing special fields  
> > > > > > > for each line e.g.: linie\_mv\_0 ... linie\_mv\_5 but this won't work when  
> > > > > > > doing m:m tree mappings - any idea what I can do or how I can limit  
> > > > > > > the facets to the one containing special characters? (I expected  
> > > > > > > facetFilter to just do this?)
> > 
> > > > > > > Best,
> > 
> > > > > > > Korbinian
> > 
> > > > > > > On 6 Mrz., 11:36, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote:
> > > > > > > 
> > > > > > > > Hi Shay,
> > 
> > > > > > > > thanks for answering. I'll try my best to explain, but don't know how  
> > > > > > > > exactly to show the output as I'm quite new to ES (coming from SOLR).
> > 
> > > > > > > > My current config now is (in elasticsearch.yml):  
> > > > > > > > index :  
> > > > > > > > analysis :  
> > > > > > > > analyzer :  
> > > > > > > > linieAnalzyer :  
> > > > > > > > type : pattern  
> > > > > > > > lowercase: false  
> > > > > > > > pattern: ", "  
> > > > > > > > flags: ""
> > 
> > > > > > > > My config for the type is as follows (from java):
> > 
> > > > > > > > String mapping = "{"produkt": {" +  
> > > > > > > > ""properties" : {" +  
> > > > > > > > ""linie\_mv" : " +  
> > > > > > > > "{ "type" : "string", " +  
> > > > > > > > ""index" : "analyzed", " +  
> > > > > > > > " "index\_analyzer": "linieAnalzyer", " +  
> > > > > > > > " "search\_analyzer": "linieAnalzyer" }" +  
> > > > > > > > "}}}";
> > 
> > > > > > > > client.admin().indices().create(new  
> > > > > > > > CreateIndexRequest(index).mapping(type, mapping)).actionGet();
> > 
> > > > > > > > and works like expected;
> > 
> > > > > > > > I then index docs containing in the linie\_mv field values like:
> > 
> > > > > > > > "\_0\_Foo, \_1\_Foo\_Bar, \_2\_Foo\_Bar\_Fii /Faa, \_3\_Foo\_Bar\_Fii /Faa\_La"
> > 
> > > > > > > > its treated at the ", " but then tokenized at all whitespaces " " and  
> > > > > > > > not as resulting single terms (after breaking it using the ", " he  
> > > > > > > > mustn't dig in deeper as he then breaks my tree;
> > 
> > > > > > > > Reason is as this is used later as a facet based upon a prefix-query,  
> > > > > > > > say e.g.:
> > 
> > > > > > > > prefix("_0_")  
> > > > > > > > facets -\> "\_0\_Foo" ; "\_0\_Foo2" etc.;
> > 
> > > > > > > > prefix"\_1\_Foo"  
> > > > > > > > -\> facets -\> "\_1\_Foo\_Bar" etc.;
> > 
> > > > > > > > So I can dynamically get the leaves of a tree and the corresponding  
> > > > > > > > doc's;
> > 
> > > > > > > > Currently I get only part right, most I also get by a prefix("\*") is  
> > > > > > > > also chunked words from inner like "/Faa\_La" here;
> > 
> > > > > > > > I hope this is now a bit more clear whats going wrong for my case.
> > 
> > > > > > > > Best,
> > 
> > > > > > > > K.
> > 
> > > > > > > > PS: Shay, thanks for your work - as a long-term compass user I really  
> > > > > > > > appreaciate it very much!
> > 
> > > > > > > > On 6 Mrz., 05:31, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > 
> > ...
> > 
> > Erfahren Sie mehr »

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:10am UTC](https://discuss.elastic.co/t/pattern-tokenization/4043/13 "2017-07-06T04:10:27Z")

</div>