# Issue with pattern analyzer

**URL:** <https://discuss.elastic.co/t/issue-with-pattern-analyzer/6688>\
**Category:** Elasticsearch\
**Created:** [February 13, 2012, 4:49pm UTC](https://discuss.elastic.co/t/issue-with-pattern-analyzer/6688 "2012-02-13T16:49:04Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Scott\_Decker](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@Scott\_Decker](https://discuss.elastic.co/u/Scott_Decker)\
**Post date:** [February 13, 2012, 4:49pm UTC](https://discuss.elastic.co/t/issue-with-pattern-analyzer/6688/1 "2012-02-13T16:49:04Z")

</div>

Hey All,  
I am trying to setup a pattern analyzer for indexes, but something  
just isn't working.

Here is what I have tried in our elasticsearch.json

"pipeDelim":{  
"type": "pattern",  
"stopwords": "_none_",  
"flags": "DOTALL",  
"lowercase":true,  
"pattern": "\|"  
},

and

"pipeDelimTest":{  
"type":"custom",  
"tokenizer":"pattern",  
"pattern":"\|",  
"filter":["lowercase"]

```
        },

```

and I pass in the following to a doc that has one of these analyzers  
Kobe Bryant|Lamar Odom

doing a search for  
Kobe Bryant (term query)

nothing comes back from a search

if I type  
bryant (term query)

then the document comes back

if I type  
kobe bryant (term query)  
nothing comes back.

so, it seems like it is just doing the lower casing, but not the pipe  
delim to separate tokens and then lower casing.

Any ideas on how to setup this up so it delimits on the pipe  
character, and then lower cases the tokens?

Thanks,  
Scott

---

<div class="post-metadata">

**Author:** ![Scott\_Decker](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@Scott\_Decker](https://discuss.elastic.co/u/Scott_Decker)\
**Post date:** [February 13, 2012, 6:36pm UTC](https://discuss.elastic.co/t/issue-with-pattern-analyzer/6688/2 "2012-02-13T18:36:46Z")

</div>

To give more clarity on this, we had been running on 17.6, and just  
upgraded to 18.7  
we re-indexed everything, and our analyzers that were setup no longer  
worked.

We set some default ones in the elasticsearch.json setup file for es  
itself.  
So, when we create indexes, anyone can use the main default ones we  
have, like this pipeDelim

however, it doesn't seem like it is working anymore.  
Here is our snippet from the json for the analyzers in  
elasticsearch.json  
"index":{  
"analysis":{  
"analyzer":{  
"pipeDelim":{  
"type": "pattern",  
"stopwords": "_none_",  
"flags": "DOTALL",  
"lowercase":true,  
"pattern": "\|"  
},  
"pipeDelimTest":{  
"type":"custom",  
"tokenizer":"pattern",  
"pattern":"\|",  
"filter":["lowercase"]

```
        }

```

}  
}  
}

neither of those seem to work now

On Feb 13, 8:49 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:

> Hey All,  
> I am trying to setup a pattern analyzer for indexes, but something  
> just isn't working.
> 
> Here is what I have tried in our elasticsearch.json
> 
> "pipeDelim":{  
> "type": "pattern",  
> "stopwords": "_none_",  
> "flags": "DOTALL",  
> "lowercase":true,  
> "pattern": "\|"  
> },
> 
> and
> 
> "pipeDelimTest":{  
> "type":"custom",  
> "tokenizer":"pattern",  
> "pattern":"\|",  
> "filter":["lowercase"]
> 
> ```
> },
> 
> ```
> 
> and I pass in the following to a doc that has one of these analyzers  
> Kobe Bryant|Lamar Odom
> 
> doing a search for  
> Kobe Bryant (term query)
> 
> nothing comes back from a search
> 
> if I type  
> bryant (term query)
> 
> then the document comes back
> 
> if I type  
> kobe bryant (term query)  
> nothing comes back.
> 
> so, it seems like it is just doing the lower casing, but not the pipe  
> delim to separate tokens and then lower casing.
> 
> Any ideas on how to setup this up so it delimits on the pipe  
> character, and then lower cases the tokens?
> 
> Thanks,  
> Scott

---

<div class="post-metadata">

**Author:** ![Scott\_Decker](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@Scott\_Decker](https://discuss.elastic.co/u/Scott_Decker)\
**Post date:** [February 13, 2012, 7:58pm UTC](https://discuss.elastic.co/t/issue-with-pattern-analyzer/6688/3 "2012-02-13T19:58:03Z")

</div>

More info.

I try this:

curl -X GET "[http://ec2loc:9200/index-to-test/\_analyze](http://ec2loc:9200/index-to-test/_analyze)?  
analyzer=pipeDelim" -d "Kobe Bryant|Lamar Odom"  
and this is the output  
{"tokens":[{"token":"kobe bryant","start\_offset":0,"end\_offset":  
11,"type":"word","position":1},{"token":"lamar odom","start\_offset":  
12,"end\_offset":22,"type":"word","position":2}]}

So, obviously the analyzer works, but the documents being put in do  
not seem to be getting analyzed correctly, nor can we search against  
those tokens.

Here is an example index mapping from one of the indexes  
entities: {

```
analyzer: pipeDelim
type: string

```

}

Anyone have any thoughts, or what else to try and test?

On Feb 13, 10:36 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:

> To give more clarity on this, we had been running on 17.6, and just  
> upgraded to 18.7  
> we re-indexed everything, and our analyzers that were setup no longer  
> worked.
> 
> We set some default ones in the elasticsearch.json setup file for es  
> itself.  
> So, when we create indexes, anyone can use the main default ones we  
> have, like this pipeDelim
> 
> however, it doesn't seem like it is working anymore.  
> Here is our snippet from the json for the analyzers in  
> elasticsearch.json  
> "index":{  
> "analysis":{  
> "analyzer":{  
> "pipeDelim":{  
> "type": "pattern",  
> "stopwords": "_none_",  
> "flags": "DOTALL",  
> "lowercase":true,  
> "pattern": "\|"  
> },  
> "pipeDelimTest":{  
> "type":"custom",  
> "tokenizer":"pattern",  
> "pattern":"\|",  
> "filter":["lowercase"]
> 
> ```
> }
> 
> ```
> 
> }  
> }  
> }
> 
> neither of those seem to work now
> 
> On Feb 13, 8:49 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:
> 
> > Hey All,  
> > I am trying to setup a pattern analyzer for indexes, but something  
> > just isn't working.
> 
> > Here is what I have tried in our elasticsearch.json
> 
> > "pipeDelim":{  
> > "type": "pattern",  
> > "stopwords": "_none_",  
> > "flags": "DOTALL",  
> > "lowercase":true,  
> > "pattern": "\|"  
> > },
> 
> > and
> 
> > "pipeDelimTest":{  
> > "type":"custom",  
> > "tokenizer":"pattern",  
> > "pattern":"\|",  
> > "filter":["lowercase"]
> 
> > ```
> > },
> > 
> > ```
> 
> > and I pass in the following to a doc that has one of these analyzers  
> > Kobe Bryant|Lamar Odom
> 
> > doing a search for  
> > Kobe Bryant (term query)
> 
> > nothing comes back from a search
> 
> > if I type  
> > bryant (term query)
> 
> > then the document comes back
> 
> > if I type  
> > kobe bryant (term query)  
> > nothing comes back.
> 
> > so, it seems like it is just doing the lower casing, but not the pipe  
> > delim to separate tokens and then lower casing.
> 
> > Any ideas on how to setup this up so it delimits on the pipe  
> > character, and then lower cases the tokens?
> 
> > Thanks,  
> > Scott

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [February 14, 2012, 9:57am UTC](https://discuss.elastic.co/t/issue-with-pattern-analyzer/6688/4 "2012-02-14T09:57:26Z")

</div>

On Mon, 2012-02-13 at 11:58 -0800, Scott Decker wrote:

> More info.

The problem, I think, is that you are providing snippets of info, rather  
than gisting a complete example demonstrating the issue.

See [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/help)

I'm guessing that the problem is not in the analysis, but how you are  
searching.

clint

> I try this:
> 
> curl -X GET "[http://ec2loc:9200/index-to-test/\_analyze](http://ec2loc:9200/index-to-test/_analyze)?  
> analyzer=pipeDelim" -d "Kobe Bryant|Lamar Odom"  
> and this is the output  
> {"tokens":[{"token":"kobe bryant","start\_offset":0,"end\_offset":  
> 11,"type":"word","position":1},{"token":"lamar odom","start\_offset":  
> 12,"end\_offset":22,"type":"word","position":2}]}
> 
> So, obviously the analyzer works, but the documents being put in do  
> not seem to be getting analyzed correctly, nor can we search against  
> those tokens.
> 
> Here is an example index mapping from one of the indexes  
> entities: {
> 
> ```
> analyzer: pipeDelim
> type: string
> 
> ```
> 
> }
> 
> Anyone have any thoughts, or what else to try and test?
> 
> On Feb 13, 10:36 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:
> 
> > To give more clarity on this, we had been running on 17.6, and just  
> > upgraded to 18.7  
> > we re-indexed everything, and our analyzers that were setup no longer  
> > worked.
> > 
> > We set some default ones in the elasticsearch.json setup file for es  
> > itself.  
> > So, when we create indexes, anyone can use the main default ones we  
> > have, like this pipeDelim
> > 
> > however, it doesn't seem like it is working anymore.  
> > Here is our snippet from the json for the analyzers in  
> > elasticsearch.json  
> > "index":{  
> > "analysis":{  
> > "analyzer":{  
> > "pipeDelim":{  
> > "type": "pattern",  
> > "stopwords": "_none_",  
> > "flags": "DOTALL",  
> > "lowercase":true,  
> > "pattern": "\|"  
> > },  
> > "pipeDelimTest":{  
> > "type":"custom",  
> > "tokenizer":"pattern",  
> > "pattern":"\|",  
> > "filter":["lowercase"]
> > 
> > ```
> > }
> > 
> > ```
> > 
> > }  
> > }  
> > }
> > 
> > neither of those seem to work now
> > 
> > On Feb 13, 8:49 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:
> > 
> > > Hey All,  
> > > I am trying to setup a pattern analyzer for indexes, but something  
> > > just isn't working.
> > 
> > > Here is what I have tried in our elasticsearch.json
> > 
> > > "pipeDelim":{  
> > > "type": "pattern",  
> > > "stopwords": "_none_",  
> > > "flags": "DOTALL",  
> > > "lowercase":true,  
> > > "pattern": "\|"  
> > > },
> > 
> > > and
> > 
> > > "pipeDelimTest":{  
> > > "type":"custom",  
> > > "tokenizer":"pattern",  
> > > "pattern":"\|",  
> > > "filter":["lowercase"]
> > 
> > > ```
> > > },
> > > 
> > > ```
> > 
> > > and I pass in the following to a doc that has one of these analyzers  
> > > Kobe Bryant|Lamar Odom
> > 
> > > doing a search for  
> > > Kobe Bryant (term query)
> > 
> > > nothing comes back from a search
> > 
> > > if I type  
> > > bryant (term query)
> > 
> > > then the document comes back
> > 
> > > if I type  
> > > kobe bryant (term query)  
> > > nothing comes back.
> > 
> > > so, it seems like it is just doing the lower casing, but not the pipe  
> > > delim to separate tokens and then lower casing.
> > 
> > > Any ideas on how to setup this up so it delimits on the pipe  
> > > character, and then lower cases the tokens?
> > 
> > > Thanks,  
> > > Scott

---

<div class="post-metadata">

**Author:** ![Scott\_Decker](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@Scott\_Decker](https://discuss.elastic.co/u/Scott_Decker)\
**Post date:** [February 14, 2012, 3:31pm UTC](https://discuss.elastic.co/t/issue-with-pattern-analyzer/6688/5 "2012-02-14T15:31:55Z")

</div>

Here ya go

> <https://gist.github.com/anonymous/1827576>

first entry is the elasticsearch.json that is what we use to start es  
the second is the mapping I have for my simple test index  
the third is the document i insert against the test index (yes, it  
goes in as type "document")  
the fourth is the query I run expecting that pipe delim works, no  
results  
the fifth is the query I run that should not work, but does

Let me know if you need more info  
Thanks,  
scott

On Feb 14, 1:57 am, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:

> On Mon, 2012-02-13 at 11:58 -0800, Scott Decker wrote:
> 
> > More info.
> 
> The problem, I think, is that you are providing snippets of info, rather  
> than gisting a complete example demonstrating the issue.
> 
> Seehttp://www.elasticsearch.org/help
> 
> I'm guessing that the problem is not in the analysis, but how you are  
> searching.
> 
> clint
> 
> > I try this:
> 
> > curl -X GET "[http://ec2loc:9200/index-to-test/\_analyze](http://ec2loc:9200/index-to-test/_analyze)?  
> > analyzer=pipeDelim" -d "Kobe Bryant|Lamar Odom"  
> > and this is the output  
> > {"tokens":[{"token":"kobe bryant","start\_offset":0,"end\_offset":  
> > 11,"type":"word","position":1},{"token":"lamar odom","start\_offset":  
> > 12,"end\_offset":22,"type":"word","position":2}]}
> 
> > So, obviously the analyzer works, but the documents being put in do  
> > not seem to be getting analyzed correctly, nor can we search against  
> > those tokens.
> 
> > Here is an example index mapping from one of the indexes  
> > entities: {
> 
> > ```
> > analyzer: pipeDelim
> > type: string
> > 
> > ```
> 
> > }
> 
> > Anyone have any thoughts, or what else to try and test?
> 
> > On Feb 13, 10:36 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:
> > 
> > > To give more clarity on this, we had been running on 17.6, and just  
> > > upgraded to 18.7  
> > > we re-indexed everything, and our analyzers that were setup no longer  
> > > worked.
> 
> > > We set some default ones in the elasticsearch.json setup file for es  
> > > itself.  
> > > So, when we create indexes, anyone can use the main default ones we  
> > > have, like this pipeDelim
> 
> > > however, it doesn't seem like it is working anymore.  
> > > Here is our snippet from the json for the analyzers in  
> > > elasticsearch.json  
> > > "index":{  
> > > "analysis":{  
> > > "analyzer":{  
> > > "pipeDelim":{  
> > > "type": "pattern",  
> > > "stopwords": "_none_",  
> > > "flags": "DOTALL",  
> > > "lowercase":true,  
> > > "pattern": "\|"  
> > > },  
> > > "pipeDelimTest":{  
> > > "type":"custom",  
> > > "tokenizer":"pattern",  
> > > "pattern":"\|",  
> > > "filter":["lowercase"]
> 
> > > ```
> > > }
> > > 
> > > ```
> > > 
> > > }  
> > > }  
> > > }
> 
> > > neither of those seem to work now
> 
> > > On Feb 13, 8:49 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:
> 
> > > > Hey All,  
> > > > I am trying to setup a pattern analyzer for indexes, but something  
> > > > just isn't working.
> 
> > > > Here is what I have tried in our elasticsearch.json
> 
> > > > "pipeDelim":{  
> > > > "type": "pattern",  
> > > > "stopwords": "_none_",  
> > > > "flags": "DOTALL",  
> > > > "lowercase":true,  
> > > > "pattern": "\|"  
> > > > },
> 
> > > > and
> 
> > > > "pipeDelimTest":{  
> > > > "type":"custom",  
> > > > "tokenizer":"pattern",  
> > > > "pattern":"\|",  
> > > > "filter":["lowercase"]
> 
> > > > ```
> > > > },
> > > > 
> > > > ```
> 
> > > > and I pass in the following to a doc that has one of these analyzers  
> > > > Kobe Bryant|Lamar Odom
> 
> > > > doing a search for  
> > > > Kobe Bryant (term query)
> 
> > > > nothing comes back from a search
> 
> > > > if I type  
> > > > bryant (term query)
> 
> > > > then the document comes back
> 
> > > > if I type  
> > > > kobe bryant (term query)  
> > > > nothing comes back.
> 
> > > > so, it seems like it is just doing the lower casing, but not the pipe  
> > > > delim to separate tokens and then lower casing.
> 
> > > > Any ideas on how to setup this up so it delimits on the pipe  
> > > > character, and then lower cases the tokens?
> 
> > > > Thanks,  
> > > > Scott

---

<div class="post-metadata">

**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\
**Post date:** [February 14, 2012, 9:15pm UTC](https://discuss.elastic.co/t/issue-with-pattern-analyzer/6688/6 "2012-02-14T21:15:55Z")

</div>

Complete, fully reproducible curl commands are the way to go for  
others to easily reproduce. Taking a glance, this looks correct, but  
an end to end reproduction will shed further light on things. Here is  
an example to reproduce a synonym bug (that is actually fixed in the  
soon to be released lucene 3.6):

> <https://gist.github.com/ppearcy/1349777>

Best Regards,  
Paul

On Feb 14, 8:31 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:

> Here ya gohttps://gist.github.com/1827576
> 
> first entry is the elasticsearch.json that is what we use to start es  
> the second is the mapping I have for my simple test index  
> the third is the document i insert against the test index (yes, it  
> goes in as type "document")  
> the fourth is the query I run expecting that pipe delim works, no  
> results  
> the fifth is the query I run that should not work, but does
> 
> Let me know if you need more info  
> Thanks,  
> scott
> 
> On Feb 14, 1:57 am, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:
> 
> > On Mon, 2012-02-13 at 11:58 -0800, Scott Decker wrote:
> > 
> > > More info.
> 
> > The problem, I think, is that you are providing snippets of info, rather  
> > than gisting a complete example demonstrating the issue.
> 
> > Seehttp://www.elasticsearch.org/help
> 
> > I'm guessing that the problem is not in the analysis, but how you are  
> > searching.
> 
> > clint
> 
> > > I try this:
> 
> > > curl -X GET "[http://ec2loc:9200/index-to-test/\_analyze](http://ec2loc:9200/index-to-test/_analyze)?  
> > > analyzer=pipeDelim" -d "Kobe Bryant|Lamar Odom"  
> > > and this is the output  
> > > {"tokens":[{"token":"kobe bryant","start\_offset":0,"end\_offset":  
> > > 11,"type":"word","position":1},{"token":"lamar odom","start\_offset":  
> > > 12,"end\_offset":22,"type":"word","position":2}]}
> 
> > > So, obviously the analyzer works, but the documents being put in do  
> > > not seem to be getting analyzed correctly, nor can we search against  
> > > those tokens.
> 
> > > Here is an example index mapping from one of the indexes  
> > > entities: {
> 
> > > ```
> > > analyzer: pipeDelim
> > > type: string
> > > 
> > > ```
> 
> > > }
> 
> > > Anyone have any thoughts, or what else to try and test?
> 
> > > On Feb 13, 10:36 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:
> > > 
> > > > To give more clarity on this, we had been running on 17.6, and just  
> > > > upgraded to 18.7  
> > > > we re-indexed everything, and our analyzers that were setup no longer  
> > > > worked.
> 
> > > > We set some default ones in the elasticsearch.json setup file for es  
> > > > itself.  
> > > > So, when we create indexes, anyone can use the main default ones we  
> > > > have, like this pipeDelim
> 
> > > > however, it doesn't seem like it is working anymore.  
> > > > Here is our snippet from the json for the analyzers in  
> > > > elasticsearch.json  
> > > > "index":{  
> > > > "analysis":{  
> > > > "analyzer":{  
> > > > "pipeDelim":{  
> > > > "type": "pattern",  
> > > > "stopwords": "_none_",  
> > > > "flags": "DOTALL",  
> > > > "lowercase":true,  
> > > > "pattern": "\|"  
> > > > },  
> > > > "pipeDelimTest":{  
> > > > "type":"custom",  
> > > > "tokenizer":"pattern",  
> > > > "pattern":"\|",  
> > > > "filter":["lowercase"]
> 
> > > > ```
> > > > }
> > > > 
> > > > ```
> > > > 
> > > > }  
> > > > }  
> > > > }
> 
> > > > neither of those seem to work now
> 
> > > > On Feb 13, 8:49 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:
> 
> > > > > Hey All,  
> > > > > I am trying to setup a pattern analyzer for indexes, but something  
> > > > > just isn't working.
> 
> > > > > Here is what I have tried in our elasticsearch.json
> 
> > > > > "pipeDelim":{  
> > > > > "type": "pattern",  
> > > > > "stopwords": "_none_",  
> > > > > "flags": "DOTALL",  
> > > > > "lowercase":true,  
> > > > > "pattern": "\|"  
> > > > > },
> 
> > > > > and
> 
> > > > > "pipeDelimTest":{  
> > > > > "type":"custom",  
> > > > > "tokenizer":"pattern",  
> > > > > "pattern":"\|",  
> > > > > "filter":["lowercase"]
> 
> > > > > ```
> > > > > },
> > > > > 
> > > > > ```
> 
> > > > > and I pass in the following to a doc that has one of these analyzers  
> > > > > Kobe Bryant|Lamar Odom
> 
> > > > > doing a search for  
> > > > > Kobe Bryant (term query)
> 
> > > > > nothing comes back from a search
> 
> > > > > if I type  
> > > > > bryant (term query)
> 
> > > > > then the document comes back
> 
> > > > > if I type  
> > > > > kobe bryant (term query)  
> > > > > nothing comes back.
> 
> > > > > so, it seems like it is just doing the lower casing, but not the pipe  
> > > > > delim to separate tokens and then lower casing.
> 
> > > > > Any ideas on how to setup this up so it delimits on the pipe  
> > > > > character, and then lower cases the tokens?
> 
> > > > > Thanks,  
> > > > > Scott

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [February 15, 2012, 10:19am UTC](https://discuss.elastic.co/t/issue-with-pattern-analyzer/6688/7 "2012-02-15T10:19:56Z")

</div>

On Tue, 2012-02-14 at 13:15 -0800, ppearcy wrote:

> Complete, fully reproducible curl commands are the way to go for  
> others to easily reproduce.

++ exactly!

You're missing a bunch of information, like what 'type' is your doc.  
You have two different types with different analysis for the same field  
(which can cause clashes depending on how you search).

Also, your pipe-delim analyzer lowercases, but then you're searching for  
"Kobe Bryant" with a term query, which does no analysis (ie it looks for  
the exact term "Kobe Bryant" while your term is "kobe bryant"

clint

> Taking a glance, this looks correct, but  
> an end to end reproduction will shed further light on things. Here is  
> an example to reproduce a synonym bug (that is actually fixed in the  
> soon to be released lucene 3.6):  
> [Synonym search failure · GitHub](https://gist.github.com/1349777)
> 
> Best Regards,  
> Paul
> 
> On Feb 14, 8:31 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:
> 
> > Here ya gohttps://gist.github.com/1827576
> > 
> > first entry is the elasticsearch.json that is what we use to start es  
> > the second is the mapping I have for my simple test index  
> > the third is the document i insert against the test index (yes, it  
> > goes in as type "document")  
> > the fourth is the query I run expecting that pipe delim works, no  
> > results  
> > the fifth is the query I run that should not work, but does
> > 
> > Let me know if you need more info  
> > Thanks,  
> > scott
> > 
> > On Feb 14, 1:57 am, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:
> > 
> > > On Mon, 2012-02-13 at 11:58 -0800, Scott Decker wrote:
> > > 
> > > > More info.
> > 
> > > The problem, I think, is that you are providing snippets of info, rather  
> > > than gisting a complete example demonstrating the issue.
> > 
> > > Seehttp://www.elasticsearch.org/help
> > 
> > > I'm guessing that the problem is not in the analysis, but how you are  
> > > searching.
> > 
> > > clint
> > 
> > > > I try this:
> > 
> > > > curl -X GET "[http://ec2loc:9200/index-to-test/\_analyze](http://ec2loc:9200/index-to-test/_analyze)?  
> > > > analyzer=pipeDelim" -d "Kobe Bryant|Lamar Odom"  
> > > > and this is the output  
> > > > {"tokens":[{"token":"kobe bryant","start\_offset":0,"end\_offset":  
> > > > 11,"type":"word","position":1},{"token":"lamar odom","start\_offset":  
> > > > 12,"end\_offset":22,"type":"word","position":2}]}
> > 
> > > > So, obviously the analyzer works, but the documents being put in do  
> > > > not seem to be getting analyzed correctly, nor can we search against  
> > > > those tokens.
> > 
> > > > Here is an example index mapping from one of the indexes  
> > > > entities: {
> > 
> > > > ```
> > > > analyzer: pipeDelim
> > > > type: string
> > > > 
> > > > ```
> > 
> > > > }
> > 
> > > > Anyone have any thoughts, or what else to try and test?
> > 
> > > > On Feb 13, 10:36 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:
> > > > 
> > > > > To give more clarity on this, we had been running on 17.6, and just  
> > > > > upgraded to 18.7  
> > > > > we re-indexed everything, and our analyzers that were setup no longer  
> > > > > worked.
> > 
> > > > > We set some default ones in the elasticsearch.json setup file for es  
> > > > > itself.  
> > > > > So, when we create indexes, anyone can use the main default ones we  
> > > > > have, like this pipeDelim
> > 
> > > > > however, it doesn't seem like it is working anymore.  
> > > > > Here is our snippet from the json for the analyzers in  
> > > > > elasticsearch.json  
> > > > > "index":{  
> > > > > "analysis":{  
> > > > > "analyzer":{  
> > > > > "pipeDelim":{  
> > > > > "type": "pattern",  
> > > > > "stopwords": "_none_",  
> > > > > "flags": "DOTALL",  
> > > > > "lowercase":true,  
> > > > > "pattern": "\|"  
> > > > > },  
> > > > > "pipeDelimTest":{  
> > > > > "type":"custom",  
> > > > > "tokenizer":"pattern",  
> > > > > "pattern":"\|",  
> > > > > "filter":["lowercase"]
> > 
> > > > > ```
> > > > > }
> > > > > 
> > > > > ```
> > > > > 
> > > > > }  
> > > > > }  
> > > > > }
> > 
> > > > > neither of those seem to work now
> > 
> > > > > On Feb 13, 8:49 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:
> > 
> > > > > > Hey All,  
> > > > > > I am trying to setup a pattern analyzer for indexes, but something  
> > > > > > just isn't working.
> > 
> > > > > > Here is what I have tried in our elasticsearch.json
> > 
> > > > > > "pipeDelim":{  
> > > > > > "type": "pattern",  
> > > > > > "stopwords": "_none_",  
> > > > > > "flags": "DOTALL",  
> > > > > > "lowercase":true,  
> > > > > > "pattern": "\|"  
> > > > > > },
> > 
> > > > > > and
> > 
> > > > > > "pipeDelimTest":{  
> > > > > > "type":"custom",  
> > > > > > "tokenizer":"pattern",  
> > > > > > "pattern":"\|",  
> > > > > > "filter":["lowercase"]
> > 
> > > > > > ```
> > > > > > },
> > > > > > 
> > > > > > ```
> > 
> > > > > > and I pass in the following to a doc that has one of these analyzers  
> > > > > > Kobe Bryant|Lamar Odom
> > 
> > > > > > doing a search for  
> > > > > > Kobe Bryant (term query)
> > 
> > > > > > nothing comes back from a search
> > 
> > > > > > if I type  
> > > > > > bryant (term query)
> > 
> > > > > > then the document comes back
> > 
> > > > > > if I type  
> > > > > > kobe bryant (term query)  
> > > > > > nothing comes back.
> > 
> > > > > > so, it seems like it is just doing the lower casing, but not the pipe  
> > > > > > delim to separate tokens and then lower casing.
> > 
> > > > > > Any ideas on how to setup this up so it delimits on the pipe  
> > > > > > character, and then lower cases the tokens?
> > 
> > > > > > Thanks,  
> > > > > > Scott

---

<div class="post-metadata">

**Author:** ![Scott\_Decker](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@Scott\_Decker](https://discuss.elastic.co/u/Scott_Decker)\
**Post date:** [February 15, 2012, 3:21pm UTC](https://discuss.elastic.co/t/issue-with-pattern-analyzer/6688/8 "2012-02-15T15:21:28Z")

</div>

Working on the curl info. We use the java api, and not the straight  
curl, so, a bit difficult to translate 1 to 1.

In response to the last comment though, the term "Kobe Bryant" should  
be correct. it is in Solr/Lucene.  
If I say search for "Kobe Bryant" the ES server should then run that  
through the analyzer for the field, and then search for it, yes? That  
should then turn it into "kobe bryant" because it will look for the  
pipe character, find none, then lowercase the token.  
Or are you saying this should be done via query\_string instead?

Scott

On Feb 15, 2:19 am, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:

> On Tue, 2012-02-14 at 13:15 -0800, ppearcy wrote:
> 
> > Complete, fully reproducible curl commands are the way to go for  
> > others to easily reproduce.
> 
> ++ exactly!
> 
> You're missing a bunch of information, like what 'type' is your doc.  
> You have two different types with different analysis for the same field  
> (which can cause clashes depending on how you search).
> 
> Also, your pipe-delim analyzer lowercases, but then you're searching for  
> "Kobe Bryant" with a term query, which does no analysis (ie it looks for  
> the exact term "Kobe Bryant" while your term is "kobe bryant"
> 
> clint
> 
> > Taking a glance, this looks correct, but  
> > an end to end reproduction will shed further light on things. Here is  
> > an example to reproduce a synonym bug (that is actually fixed in the  
> > soon to be released lucene 3.6):  
> > [Synonym search failure · GitHub](https://gist.github.com/1349777)
> 
> > Best Regards,  
> > Paul
> 
> > On Feb 14, 8:31 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:
> > 
> > > Here ya gohttps://gist.github.com/1827576
> 
> > > first entry is the elasticsearch.json that is what we use to start es  
> > > the second is the mapping I have for my simple test index  
> > > the third is the document i insert against the test index (yes, it  
> > > goes in as type "document")  
> > > the fourth is the query I run expecting that pipe delim works, no  
> > > results  
> > > the fifth is the query I run that should not work, but does
> 
> > > Let me know if you need more info  
> > > Thanks,  
> > > scott
> 
> > > On Feb 14, 1:57 am, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:
> 
> > > > On Mon, 2012-02-13 at 11:58 -0800, Scott Decker wrote:
> > > > 
> > > > > More info.
> 
> > > > The problem, I think, is that you are providing snippets of info, rather  
> > > > than gisting a complete example demonstrating the issue.
> 
> > > > Seehttp://www.elasticsearch.org/help
> 
> > > > I'm guessing that the problem is not in the analysis, but how you are  
> > > > searching.
> 
> > > > clint
> 
> > > > > I try this:
> 
> > > > > curl -X GET "[http://ec2loc:9200/index-to-test/\_analyze](http://ec2loc:9200/index-to-test/_analyze)?  
> > > > > analyzer=pipeDelim" -d "Kobe Bryant|Lamar Odom"  
> > > > > and this is the output  
> > > > > {"tokens":[{"token":"kobe bryant","start\_offset":0,"end\_offset":  
> > > > > 11,"type":"word","position":1},{"token":"lamar odom","start\_offset":  
> > > > > 12,"end\_offset":22,"type":"word","position":2}]}
> 
> > > > > So, obviously the analyzer works, but the documents being put in do  
> > > > > not seem to be getting analyzed correctly, nor can we search against  
> > > > > those tokens.
> 
> > > > > Here is an example index mapping from one of the indexes  
> > > > > entities: {
> 
> > > > > ```
> > > > > analyzer: pipeDelim
> > > > > type: string
> > > > > 
> > > > > ```
> 
> > > > > }
> 
> > > > > Anyone have any thoughts, or what else to try and test?
> 
> > > > > On Feb 13, 10:36 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:
> > > > > 
> > > > > > To give more clarity on this, we had been running on 17.6, and just  
> > > > > > upgraded to 18.7  
> > > > > > we re-indexed everything, and our analyzers that were setup no longer  
> > > > > > worked.
> 
> > > > > > We set some default ones in the elasticsearch.json setup file for es  
> > > > > > itself.  
> > > > > > So, when we create indexes, anyone can use the main default ones we  
> > > > > > have, like this pipeDelim
> 
> > > > > > however, it doesn't seem like it is working anymore.  
> > > > > > Here is our snippet from the json for the analyzers in  
> > > > > > elasticsearch.json  
> > > > > > "index":{  
> > > > > > "analysis":{  
> > > > > > "analyzer":{  
> > > > > > "pipeDelim":{  
> > > > > > "type": "pattern",  
> > > > > > "stopwords": "_none_",  
> > > > > > "flags": "DOTALL",  
> > > > > > "lowercase":true,  
> > > > > > "pattern": "\|"  
> > > > > > },  
> > > > > > "pipeDelimTest":{  
> > > > > > "type":"custom",  
> > > > > > "tokenizer":"pattern",  
> > > > > > "pattern":"\|",  
> > > > > > "filter":["lowercase"]
> 
> > > > > > ```
> > > > > > }
> > > > > > 
> > > > > > ```
> > > > > > 
> > > > > > }  
> > > > > > }  
> > > > > > }
> 
> > > > > > neither of those seem to work now
> 
> > > > > > On Feb 13, 8:49 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:
> 
> > > > > > > Hey All,  
> > > > > > > I am trying to setup a pattern analyzer for indexes, but something  
> > > > > > > just isn't working.
> 
> > > > > > > Here is what I have tried in our elasticsearch.json
> 
> > > > > > > "pipeDelim":{  
> > > > > > > "type": "pattern",  
> > > > > > > "stopwords": "_none_",  
> > > > > > > "flags": "DOTALL",  
> > > > > > > "lowercase":true,  
> > > > > > > "pattern": "\|"  
> > > > > > > },
> 
> > > > > > > and
> 
> > > > > > > "pipeDelimTest":{  
> > > > > > > "type":"custom",  
> > > > > > > "tokenizer":"pattern",  
> > > > > > > "pattern":"\|",  
> > > > > > > "filter":["lowercase"]
> 
> > > > > > > ```
> > > > > > > },
> > > > > > > 
> > > > > > > ```
> 
> > > > > > > and I pass in the following to a doc that has one of these analyzers  
> > > > > > > Kobe Bryant|Lamar Odom
> 
> > > > > > > doing a search for  
> > > > > > > Kobe Bryant (term query)
> 
> > > > > > > nothing comes back from a search
> 
> > > > > > > if I type  
> > > > > > > bryant (term query)
> 
> > > > > > > then the document comes back
> 
> > > > > > > if I type  
> > > > > > > kobe bryant (term query)  
> > > > > > > nothing comes back.
> 
> > > > > > > so, it seems like it is just doing the lower casing, but not the pipe  
> > > > > > > delim to separate tokens and then lower casing.
> 
> > > > > > > Any ideas on how to setup this up so it delimits on the pipe  
> > > > > > > character, and then lower cases the tokens?
> 
> > > > > > > Thanks,  
> > > > > > > Scott

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [February 15, 2012, 3:31pm UTC](https://discuss.elastic.co/t/issue-with-pattern-analyzer/6688/9 "2012-02-15T15:31:51Z")

</div>

> In response to the last comment though, the term "Kobe Bryant" should  
> be correct. it is in Solr/Lucene.  
> If I say search for "Kobe Bryant" the ES server should then run that  
> through the analyzer for the field, and then search for it, yes? That  
> should then turn it into "kobe bryant" because it will look for the  
> pipe character, find none, then lowercase the token.  
> Or are you saying this should be done via query\_string instead?

The term query doesn't analyze. text/query\_string/field queries do  
analyze.

c

> Scott
> 
> On Feb 15, 2:19 am, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:
> 
> > On Tue, 2012-02-14 at 13:15 -0800, ppearcy wrote:
> > 
> > > Complete, fully reproducible curl commands are the way to go for  
> > > others to easily reproduce.
> > 
> > ++ exactly!
> > 
> > You're missing a bunch of information, like what 'type' is your doc.  
> > You have two different types with different analysis for the same field  
> > (which can cause clashes depending on how you search).
> > 
> > Also, your pipe-delim analyzer lowercases, but then you're searching for  
> > "Kobe Bryant" with a term query, which does no analysis (ie it looks for  
> > the exact term "Kobe Bryant" while your term is "kobe bryant"
> > 
> > clint
> > 
> > > Taking a glance, this looks correct, but  
> > > an end to end reproduction will shed further light on things. Here is  
> > > an example to reproduce a synonym bug (that is actually fixed in the  
> > > soon to be released lucene 3.6):  
> > > [Synonym search failure · GitHub](https://gist.github.com/1349777)
> > 
> > > Best Regards,  
> > > Paul
> > 
> > > On Feb 14, 8:31 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:
> > > 
> > > > Here ya gohttps://gist.github.com/1827576
> > 
> > > > first entry is the elasticsearch.json that is what we use to start es  
> > > > the second is the mapping I have for my simple test index  
> > > > the third is the document i insert against the test index (yes, it  
> > > > goes in as type "document")  
> > > > the fourth is the query I run expecting that pipe delim works, no  
> > > > results  
> > > > the fifth is the query I run that should not work, but does
> > 
> > > > Let me know if you need more info  
> > > > Thanks,  
> > > > scott
> > 
> > > > On Feb 14, 1:57 am, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:
> > 
> > > > > On Mon, 2012-02-13 at 11:58 -0800, Scott Decker wrote:
> > > > > 
> > > > > > More info.
> > 
> > > > > The problem, I think, is that you are providing snippets of info, rather  
> > > > > than gisting a complete example demonstrating the issue.
> > 
> > > > > Seehttp://www.elasticsearch.org/help
> > 
> > > > > I'm guessing that the problem is not in the analysis, but how you are  
> > > > > searching.
> > 
> > > > > clint
> > 
> > > > > > I try this:
> > 
> > > > > > curl -X GET "[http://ec2loc:9200/index-to-test/\_analyze](http://ec2loc:9200/index-to-test/_analyze)?  
> > > > > > analyzer=pipeDelim" -d "Kobe Bryant|Lamar Odom"  
> > > > > > and this is the output  
> > > > > > {"tokens":[{"token":"kobe bryant","start\_offset":0,"end\_offset":  
> > > > > > 11,"type":"word","position":1},{"token":"lamar odom","start\_offset":  
> > > > > > 12,"end\_offset":22,"type":"word","position":2}]}
> > 
> > > > > > So, obviously the analyzer works, but the documents being put in do  
> > > > > > not seem to be getting analyzed correctly, nor can we search against  
> > > > > > those tokens.
> > 
> > > > > > Here is an example index mapping from one of the indexes  
> > > > > > entities: {
> > 
> > > > > > ```
> > > > > > analyzer: pipeDelim
> > > > > > type: string
> > > > > > 
> > > > > > ```
> > 
> > > > > > }
> > 
> > > > > > Anyone have any thoughts, or what else to try and test?
> > 
> > > > > > On Feb 13, 10:36 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:
> > > > > > 
> > > > > > > To give more clarity on this, we had been running on 17.6, and just  
> > > > > > > upgraded to 18.7  
> > > > > > > we re-indexed everything, and our analyzers that were setup no longer  
> > > > > > > worked.
> > 
> > > > > > > We set some default ones in the elasticsearch.json setup file for es  
> > > > > > > itself.  
> > > > > > > So, when we create indexes, anyone can use the main default ones we  
> > > > > > > have, like this pipeDelim
> > 
> > > > > > > however, it doesn't seem like it is working anymore.  
> > > > > > > Here is our snippet from the json for the analyzers in  
> > > > > > > elasticsearch.json  
> > > > > > > "index":{  
> > > > > > > "analysis":{  
> > > > > > > "analyzer":{  
> > > > > > > "pipeDelim":{  
> > > > > > > "type": "pattern",  
> > > > > > > "stopwords": "_none_",  
> > > > > > > "flags": "DOTALL",  
> > > > > > > "lowercase":true,  
> > > > > > > "pattern": "\|"  
> > > > > > > },  
> > > > > > > "pipeDelimTest":{  
> > > > > > > "type":"custom",  
> > > > > > > "tokenizer":"pattern",  
> > > > > > > "pattern":"\|",  
> > > > > > > "filter":["lowercase"]
> > 
> > > > > > > ```
> > > > > > > }
> > > > > > > 
> > > > > > > ```
> > > > > > > 
> > > > > > > }  
> > > > > > > }  
> > > > > > > }
> > 
> > > > > > > neither of those seem to work now
> > 
> > > > > > > On Feb 13, 8:49 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:
> > 
> > > > > > > > Hey All,  
> > > > > > > > I am trying to setup a pattern analyzer for indexes, but something  
> > > > > > > > just isn't working.
> > 
> > > > > > > > Here is what I have tried in our elasticsearch.json
> > 
> > > > > > > > "pipeDelim":{  
> > > > > > > > "type": "pattern",  
> > > > > > > > "stopwords": "_none_",  
> > > > > > > > "flags": "DOTALL",  
> > > > > > > > "lowercase":true,  
> > > > > > > > "pattern": "\|"  
> > > > > > > > },
> > 
> > > > > > > > and
> > 
> > > > > > > > "pipeDelimTest":{  
> > > > > > > > "type":"custom",  
> > > > > > > > "tokenizer":"pattern",  
> > > > > > > > "pattern":"\|",  
> > > > > > > > "filter":["lowercase"]
> > 
> > > > > > > > ```
> > > > > > > > },
> > > > > > > > 
> > > > > > > > ```
> > 
> > > > > > > > and I pass in the following to a doc that has one of these analyzers  
> > > > > > > > Kobe Bryant|Lamar Odom
> > 
> > > > > > > > doing a search for  
> > > > > > > > Kobe Bryant (term query)
> > 
> > > > > > > > nothing comes back from a search
> > 
> > > > > > > > if I type  
> > > > > > > > bryant (term query)
> > 
> > > > > > > > then the document comes back
> > 
> > > > > > > > if I type  
> > > > > > > > kobe bryant (term query)  
> > > > > > > > nothing comes back.
> > 
> > > > > > > > so, it seems like it is just doing the lower casing, but not the pipe  
> > > > > > > > delim to separate tokens and then lower casing.
> > 
> > > > > > > > Any ideas on how to setup this up so it delimits on the pipe  
> > > > > > > > character, and then lower cases the tokens?
> > 
> > > > > > > > Thanks,  
> > > > > > > > Scott

---

<div class="post-metadata">

**Author:** ![Scott\_Decker](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@Scott\_Decker](https://discuss.elastic.co/u/Scott_Decker)\
**Post date:** [February 15, 2012, 5:34pm UTC](https://discuss.elastic.co/t/issue-with-pattern-analyzer/6688/10 "2012-02-15T17:34:26Z")

</div>

Well jigger my timbers.  
That was it.  
using the query string works fine, but the term one did not.

Hmm.. I would recommend updating the docs on that, as that wasn't  
quite clear.  
Maybe something on this page here:

> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.

that calls out  
query\_string is handled by ES when you search  
all other term|prefex and other queries must be tokenized by your  
application before sending to ES

On Feb 15, 7:31 am, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:

> > In response to the last comment though, the term "Kobe Bryant" should  
> > be correct. it is in Solr/Lucene.  
> > If I say search for "Kobe Bryant" the ES server should then run that  
> > through the analyzer for the field, and then search for it, yes? That  
> > should then turn it into "kobe bryant" because it will look for the  
> > pipe character, find none, then lowercase the token.  
> > Or are you saying this should be done via query\_string instead?
> 
> The term query doesn't analyze. text/query\_string/field queries do  
> analyze.
> 
> c
> 
> > Scott
> 
> > On Feb 15, 2:19 am, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:
> > 
> > > On Tue, 2012-02-14 at 13:15 -0800, ppearcy wrote:
> > > 
> > > > Complete, fully reproducible curl commands are the way to go for  
> > > > others to easily reproduce.
> 
> > > ++ exactly!
> 
> > > You're missing a bunch of information, like what 'type' is your doc.  
> > > You have two different types with different analysis for the same field  
> > > (which can cause clashes depending on how you search).
> 
> > > Also, your pipe-delim analyzer lowercases, but then you're searching for  
> > > "Kobe Bryant" with a term query, which does no analysis (ie it looks for  
> > > the exact term "Kobe Bryant" while your term is "kobe bryant"
> 
> > > clint
> 
> > > > Taking a glance, this looks correct, but  
> > > > an end to end reproduction will shed further light on things. Here is  
> > > > an example to reproduce a synonym bug (that is actually fixed in the  
> > > > soon to be released lucene 3.6):  
> > > > [Synonym search failure · GitHub](https://gist.github.com/1349777)
> 
> > > > Best Regards,  
> > > > Paul
> 
> > > > On Feb 14, 8:31 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:
> > > > 
> > > > > Here ya gohttps://gist.github.com/1827576
> 
> > > > > first entry is the elasticsearch.json that is what we use to start es  
> > > > > the second is the mapping I have for my simple test index  
> > > > > the third is the document i insert against the test index (yes, it  
> > > > > goes in as type "document")  
> > > > > the fourth is the query I run expecting that pipe delim works, no  
> > > > > results  
> > > > > the fifth is the query I run that should not work, but does
> 
> > > > > Let me know if you need more info  
> > > > > Thanks,  
> > > > > scott
> 
> > > > > On Feb 14, 1:57 am, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:
> 
> > > > > > On Mon, 2012-02-13 at 11:58 -0800, Scott Decker wrote:
> > > > > > 
> > > > > > > More info.
> 
> > > > > > The problem, I think, is that you are providing snippets of info, rather  
> > > > > > than gisting a complete example demonstrating the issue.
> 
> > > > > > Seehttp://www.elasticsearch.org/help
> 
> > > > > > I'm guessing that the problem is not in the analysis, but how you are  
> > > > > > searching.
> 
> > > > > > clint
> 
> > > > > > > I try this:
> 
> > > > > > > curl -X GET "[http://ec2loc:9200/index-to-test/\_analyze](http://ec2loc:9200/index-to-test/_analyze)?  
> > > > > > > analyzer=pipeDelim" -d "Kobe Bryant|Lamar Odom"  
> > > > > > > and this is the output  
> > > > > > > {"tokens":[{"token":"kobe bryant","start\_offset":0,"end\_offset":  
> > > > > > > 11,"type":"word","position":1},{"token":"lamar odom","start\_offset":  
> > > > > > > 12,"end\_offset":22,"type":"word","position":2}]}
> 
> > > > > > > So, obviously the analyzer works, but the documents being put in do  
> > > > > > > not seem to be getting analyzed correctly, nor can we search against  
> > > > > > > those tokens.
> 
> > > > > > > Here is an example index mapping from one of the indexes  
> > > > > > > entities: {
> 
> > > > > > > ```
> > > > > > > analyzer: pipeDelim
> > > > > > > type: string
> > > > > > > 
> > > > > > > ```
> 
> > > > > > > }
> 
> > > > > > > Anyone have any thoughts, or what else to try and test?
> 
> > > > > > > On Feb 13, 10:36 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:
> > > > > > > 
> > > > > > > > To give more clarity on this, we had been running on 17.6, and just  
> > > > > > > > upgraded to 18.7  
> > > > > > > > we re-indexed everything, and our analyzers that were setup no longer  
> > > > > > > > worked.
> 
> > > > > > > > We set some default ones in the elasticsearch.json setup file for es  
> > > > > > > > itself.  
> > > > > > > > So, when we create indexes, anyone can use the main default ones we  
> > > > > > > > have, like this pipeDelim
> 
> > > > > > > > however, it doesn't seem like it is working anymore.  
> > > > > > > > Here is our snippet from the json for the analyzers in  
> > > > > > > > elasticsearch.json  
> > > > > > > > "index":{  
> > > > > > > > "analysis":{  
> > > > > > > > "analyzer":{  
> > > > > > > > "pipeDelim":{  
> > > > > > > > "type": "pattern",  
> > > > > > > > "stopwords": "_none_",  
> > > > > > > > "flags": "DOTALL",  
> > > > > > > > "lowercase":true,  
> > > > > > > > "pattern": "\|"  
> > > > > > > > },  
> > > > > > > > "pipeDelimTest":{  
> > > > > > > > "type":"custom",  
> > > > > > > > "tokenizer":"pattern",  
> > > > > > > > "pattern":"\|",  
> > > > > > > > "filter":["lowercase"]
> 
> > > > > > > > ```
> > > > > > > > }
> > > > > > > > 
> > > > > > > > ```
> > > > > > > > 
> > > > > > > > }  
> > > > > > > > }  
> > > > > > > > }
> 
> > > > > > > > neither of those seem to work now
> 
> > > > > > > > On Feb 13, 8:49 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:
> 
> > > > > > > > > Hey All,  
> > > > > > > > > I am trying to setup a pattern analyzer for indexes, but something  
> > > > > > > > > just isn't working.
> 
> > > > > > > > > Here is what I have tried in our elasticsearch.json
> 
> > > > > > > > > "pipeDelim":{  
> > > > > > > > > "type": "pattern",  
> > > > > > > > > "stopwords": "_none_",  
> > > > > > > > > "flags": "DOTALL",  
> > > > > > > > > "lowercase":true,  
> > > > > > > > > "pattern": "\|"  
> > > > > > > > > },
> 
> > > > > > > > > and
> 
> > > > > > > > > "pipeDelimTest":{  
> > > > > > > > > "type":"custom",  
> > > > > > > > > "tokenizer":"pattern",  
> > > > > > > > > "pattern":"\|",  
> > > > > > > > > "filter":["lowercase"]
> 
> > > > > > > > > ```
> > > > > > > > > },
> > > > > > > > > 
> > > > > > > > > ```
> 
> > > > > > > > > and I pass in the following to a doc that has one of these analyzers  
> > > > > > > > > Kobe Bryant|Lamar Odom
> 
> > > > > > > > > doing a search for  
> > > > > > > > > Kobe Bryant (term query)
> 
> > > > > > > > > nothing comes back from a search
> 
> > > > > > > > > if I type  
> > > > > > > > > bryant (term query)
> 
> > > > > > > > > then the document comes back
> 
> > > > > > > > > if I type  
> > > > > > > > > kobe bryant (term query)  
> > > > > > > > > nothing comes back.
> 
> > > > > > > > > so, it seems like it is just doing the lower casing, but not the pipe  
> > > > > > > > > delim to separate tokens and then lower casing.
> 
> > > > > > > > > Any ideas on how to setup this up so it delimits on the pipe  
> > > > > > > > > character, and then lower cases the tokens?
> 
> > > > > > > > > Thanks,  
> > > > > > > > > Scott

---

<div class="post-metadata">

**Author:** ![Ted\_Hromadka](https://avatars.discourse-cdn.com/v4/letter/t/dc4da7/32.png) [@Ted\_Hromadka](https://discuss.elastic.co/u/Ted_Hromadka)\
**Post date:** [February 22, 2012, 10:17pm UTC](https://discuss.elastic.co/t/issue-with-pattern-analyzer/6688/11 "2012-02-22T22:17:57Z")

</div>

agreed... the docs do mention what is not analyzed, but could be  
louder about tokenizing

> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.

> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.

On Feb 15, 9:34 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:

> Well jigger my timbers.  
> That was it.  
> using the query string works fine, but the term one did not.
> 
> Hmm.. I would recommend updating the docs on that, as that wasn't  
> quite clear.  
> Maybe something on this page here:[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/query-dsl/)
> 
> that calls out  
> query\_string is handled by ES when you search  
> all other term|prefex and other queries must be tokenized by your  
> application before sending to ES
> 
> On Feb 15, 7:31 am, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:
> 
> > > In response to the last comment though, the term "Kobe Bryant" should  
> > > be correct. it is in Solr/Lucene.  
> > > If I say search for "Kobe Bryant" the ES server should then run that  
> > > through the analyzer for the field, and then search for it, yes? That  
> > > should then turn it into "kobe bryant" because it will look for the  
> > > pipe character, find none, then lowercase the token.  
> > > Or are you saying this should be done via query\_string instead?
> 
> > The term query doesn't analyze. text/query\_string/field queries do  
> > analyze.
> 
> > c
> 
> > > Scott
> 
> > > On Feb 15, 2:19 am, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:
> > > 
> > > > On Tue, 2012-02-14 at 13:15 -0800, ppearcy wrote:
> > > > 
> > > > > Complete, fully reproducible curl commands are the way to go for  
> > > > > others to easily reproduce.
> 
> > > > ++ exactly!
> 
> > > > You're missing a bunch of information, like what 'type' is your doc.  
> > > > You have two different types with different analysis for the same field  
> > > > (which can cause clashes depending on how you search).
> 
> > > > Also, your pipe-delim analyzer lowercases, but then you're searching for  
> > > > "Kobe Bryant" with a term query, which does no analysis (ie it looks for  
> > > > the exact term "Kobe Bryant" while your term is "kobe bryant"
> 
> > > > clint
> 
> > > > > Taking a glance, this looks correct, but  
> > > > > an end to end reproduction will shed further light on things. Here is  
> > > > > an example to reproduce a synonym bug (that is actually fixed in the  
> > > > > soon to be released lucene 3.6):  
> > > > > [Synonym search failure · GitHub](https://gist.github.com/1349777)
> 
> > > > > Best Regards,  
> > > > > Paul
> 
> > > > > On Feb 14, 8:31 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:
> > > > > 
> > > > > > Here ya gohttps://gist.github.com/1827576
> 
> > > > > > first entry is the elasticsearch.json that is what we use to start es  
> > > > > > the second is the mapping I have for my simple test index  
> > > > > > the third is the document i insert against the test index (yes, it  
> > > > > > goes in as type "document")  
> > > > > > the fourth is the query I run expecting that pipe delim works, no  
> > > > > > results  
> > > > > > the fifth is the query I run that should not work, but does
> 
> > > > > > Let me know if you need more info  
> > > > > > Thanks,  
> > > > > > scott
> 
> > > > > > On Feb 14, 1:57 am, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:
> 
> > > > > > > On Mon, 2012-02-13 at 11:58 -0800, Scott Decker wrote:
> > > > > > > 
> > > > > > > > More info.
> 
> > > > > > > The problem, I think, is that you are providing snippets of info, rather  
> > > > > > > than gisting a complete example demonstrating the issue.
> 
> > > > > > > Seehttp://www.elasticsearch.org/help
> 
> > > > > > > I'm guessing that the problem is not in the analysis, but how you are  
> > > > > > > searching.
> 
> > > > > > > clint
> 
> > > > > > > > I try this:
> 
> > > > > > > > curl -X GET "[http://ec2loc:9200/index-to-test/\_analyze](http://ec2loc:9200/index-to-test/_analyze)?  
> > > > > > > > analyzer=pipeDelim" -d "Kobe Bryant|Lamar Odom"  
> > > > > > > > and this is the output  
> > > > > > > > {"tokens":[{"token":"kobe bryant","start\_offset":0,"end\_offset":  
> > > > > > > > 11,"type":"word","position":1},{"token":"lamar odom","start\_offset":  
> > > > > > > > 12,"end\_offset":22,"type":"word","position":2}]}
> 
> > > > > > > > So, obviously the analyzer works, but the documents being put in do  
> > > > > > > > not seem to be getting analyzed correctly, nor can we search against  
> > > > > > > > those tokens.
> 
> > > > > > > > Here is an example index mapping from one of the indexes  
> > > > > > > > entities: {
> 
> > > > > > > > ```
> > > > > > > > analyzer: pipeDelim
> > > > > > > > type: string
> > > > > > > > 
> > > > > > > > ```
> 
> > > > > > > > }
> 
> > > > > > > > Anyone have any thoughts, or what else to try and test?
> 
> > > > > > > > On Feb 13, 10:36 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:
> > > > > > > > 
> > > > > > > > > To give more clarity on this, we had been running on 17.6, and just  
> > > > > > > > > upgraded to 18.7  
> > > > > > > > > we re-indexed everything, and our analyzers that were setup no longer  
> > > > > > > > > worked.
> 
> > > > > > > > > We set some default ones in the elasticsearch.json setup file for es  
> > > > > > > > > itself.  
> > > > > > > > > So, when we create indexes, anyone can use the main default ones we  
> > > > > > > > > have, like this pipeDelim
> 
> > > > > > > > > however, it doesn't seem like it is working anymore.  
> > > > > > > > > Here is our snippet from the json for the analyzers in  
> > > > > > > > > elasticsearch.json  
> > > > > > > > > "index":{  
> > > > > > > > > "analysis":{  
> > > > > > > > > "analyzer":{  
> > > > > > > > > "pipeDelim":{  
> > > > > > > > > "type": "pattern",  
> > > > > > > > > "stopwords": "_none_",  
> > > > > > > > > "flags": "DOTALL",  
> > > > > > > > > "lowercase":true,  
> > > > > > > > > "pattern": "\|"  
> > > > > > > > > },  
> > > > > > > > > "pipeDelimTest":{  
> > > > > > > > > "type":"custom",  
> > > > > > > > > "tokenizer":"pattern",  
> > > > > > > > > "pattern":"\|",  
> > > > > > > > > "filter":["lowercase"]
> 
> > > > > > > > > ```
> > > > > > > > > }
> > > > > > > > > 
> > > > > > > > > ```
> > > > > > > > > 
> > > > > > > > > }  
> > > > > > > > > }  
> > > > > > > > > }
> 
> > > > > > > > > neither of those seem to work now
> 
> > > > > > > > > On Feb 13, 8:49 am, Scott Decker [sc...@publishthis.com](mailto:sc...@publishthis.com) wrote:
> 
> > > > > > > > > > Hey All,  
> > > > > > > > > > I am trying to setup a pattern analyzer for indexes, but something  
> > > > > > > > > > just isn't working.
> 
> > > > > > > > > > Here is what I have tried in our elasticsearch.json
> 
> > > > > > > > > > "pipeDelim":{  
> > > > > > > > > > "type": "pattern",  
> > > > > > > > > > "stopwords": "_none_",  
> > > > > > > > > > "flags": "DOTALL",  
> > > > > > > > > > "lowercase":true,  
> > > > > > > > > > "pattern": "\|"  
> > > > > > > > > > },
> 
> > > > > > > > > > and
> 
> > > > > > > > > > "pipeDelimTest":{  
> > > > > > > > > > "type":"custom",  
> > > > > > > > > > "tokenizer":"pattern",  
> > > > > > > > > > "pattern":"\|",  
> > > > > > > > > > "filter":["lowercase"]
> 
> > > > > > > > > > ```
> > > > > > > > > > },
> > > > > > > > > > 
> > > > > > > > > > ```
> 
> > > > > > > > > > and I pass in the following to a doc that has one of these analyzers  
> > > > > > > > > > Kobe Bryant|Lamar Odom
> 
> > > > > > > > > > doing a search for  
> > > > > > > > > > Kobe Bryant (term query)
> 
> > > > > > > > > > nothing comes back from a search
> 
> > > > > > > > > > if I type  
> > > > > > > > > > bryant (term query)
> 
> > > > > > > > > > then the document comes back
> 
> > > > > > > > > > if I type  
> > > > > > > > > > kobe bryant (term query)  
> > > > > > > > > > nothing comes back.
> 
> > > > > > > > > > so, it seems like it is just doing the lower casing, but not the pipe  
> > > > > > > > > > delim to separate tokens and then lower casing.
> 
> > > > > > > > > > Any ideas on how to setup this up so it delimits on the pipe  
> > > > > > > > > > character, and then lower cases the tokens?
> 
> > > > > > > > > > Thanks,  
> > > > > > > > > > Scott

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:38am UTC](https://discuss.elastic.co/t/issue-with-pattern-analyzer/6688/12 "2017-07-06T03:38:30Z")

</div>