# Removing whitespace around a delimiter in a custom anaylzer

**URL:** <https://discuss.elastic.co/t/removing-whitespace-around-a-delimiter-in-a-custom-anaylzer/6624>\
**Category:** Elasticsearch\
**Created:** [February 7, 2012, 4:25pm UTC](https://discuss.elastic.co/t/removing-whitespace-around-a-delimiter-in-a-custom-anaylzer/6624 "2012-02-07T16:25:38Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rick\_Thomas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rick_thomas/32/2993_2.png) [@Rick\_Thomas](https://discuss.elastic.co/u/Rick_Thomas)\
**Post date:** [February 7, 2012, 4:25pm UTC](https://discuss.elastic.co/t/removing-whitespace-around-a-delimiter-in-a-custom-anaylzer/6624/1 "2012-02-07T16:25:38Z")

</div>

I'm having difficulty with a custom analyzer. I have a field in my  
index that looks like this [I am a token, I'm a token too, Tokenize me,This is a token,Tokenize me]

Instead of creating term facets based on spaces, I want to create term  
facets based on commas. I also need to remove any whitespace around  
the comma. Here is my analyzer:

"analysis":{"analyzer":{"comma":{"type":"pattern","pattern":"\s\*,  
\s\*"}}}

It works in that it tokenizes string based on commas, but it is  
including trailing and leading whitespace in the tokens. I need to  
get rid of that whitespace. The regex I'm using is supposed to do  
that (it should pick up 0 to n spaces on either side of the comma),  
but it is not doing that.

Any thoughts on how I can force the regex engine to be greedier in its  
analysis?

Thanks,

Rick

---

<div class="post-metadata">

**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\
**Post date:** [February 7, 2012, 8:15pm UTC](https://discuss.elastic.co/t/removing-whitespace-around-a-delimiter-in-a-custom-anaylzer/6624/2 "2012-02-07T20:15:42Z")

</div>

probably have a look into worddelimiter filter

Peter.

On 7 Feb., 17:25, Rick Thomas [rtho...@igodigital.com](mailto:rtho...@igodigital.com) wrote:

> I'm having difficulty with a custom analyzer. I have a field in my  
> index that looks like this [I am a token, I'm a token too, Tokenize me,This is a token,Tokenize me]
> 
> Instead of creating term facets based on spaces, I want to create term  
> facets based on commas. I also need to remove any whitespace around  
> the comma. Here is my analyzer:
> 
> "analysis":{"analyzer":{"comma":{"type":"pattern","pattern":"\s\*,  
> \s\*"}}}
> 
> It works in that it tokenizes string based on commas, but it is  
> including trailing and leading whitespace in the tokens. I need to  
> get rid of that whitespace. The regex I'm using is supposed to do  
> that (it should pick up 0 to n spaces on either side of the comma),  
> but it is not doing that.
> 
> Any thoughts on how I can force the regex engine to be greedier in its  
> analysis?
> 
> Thanks,
> 
> Rick

---

<div class="post-metadata">

**Author:** ![Rick\_Thomas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rick_thomas/32/2993_2.png) [@Rick\_Thomas](https://discuss.elastic.co/u/Rick_Thomas)\
**Post date:** [February 8, 2012, 1:14am UTC](https://discuss.elastic.co/t/removing-whitespace-around-a-delimiter-in-a-custom-anaylzer/6624/3 "2012-02-08T01:14:55Z")

</div>

That appears to do the opposite of what I need.

On Feb 7, 3:15 pm, Karussell [tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com) wrote:

> probably have a look into worddelimiter filter
> 
> Peter.
> 
> On 7 Feb., 17:25, Rick Thomas [rtho...@igodigital.com](mailto:rtho...@igodigital.com) wrote:
> 
> > I'm having difficulty with a custom analyzer. I have a field in my  
> > index that looks like this [I am a token, I'm a token too, Tokenize me,This is a token,Tokenize me]
> 
> > Instead of creating term facets based on spaces, I want to create term  
> > facets based on commas. I also need to remove anywhitespacearound  
> > the comma. Here is my analyzer:
> 
> > "analysis":{"analyzer":{"comma":{"type":"pattern","pattern":"\s\*,  
> > \s\*"}}}
> 
> > It works in that it tokenizes string based on commas, but it is  
> > including trailing and leadingwhitespacein the tokens. I need to  
> > get rid of thatwhitespace. The regex I'm using is supposed to do  
> > that (it should pick up 0 to n spaces on either side of the comma),  
> > but it is not doing that.
> 
> > Any thoughts on how I can force the regex engine to be greedier in its  
> > analysis?
> 
> > Thanks,
> 
> > Rick

---

<div class="post-metadata">

**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\
**Post date:** [February 8, 2012, 1:26pm UTC](https://discuss.elastic.co/t/removing-whitespace-around-a-delimiter-in-a-custom-anaylzer/6624/4 "2012-02-08T13:26:22Z")

</div>

> That appears to do the opposite of what I need.

I think you can hack this worddelimiter thing a lot. E.g. overwriting  
the comma char to be recognized as SUBWORD\_DELIM (see type\_table)

> I have a field in my index that looks like this

do you mean the field or the original data?

> Any thoughts on how I can force the regex engine to be greedier in its analysis?

No idea, I'm avoiding regex when and where I can 🙂

so I would do this via a custom WhitespaceTokenizer which overwrites  
isTokenChar

Peter.

---

<div class="post-metadata">

**Author:** ![Rick\_Thomas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rick_thomas/32/2993_2.png) [@Rick\_Thomas](https://discuss.elastic.co/u/Rick_Thomas)\
**Post date:** [February 8, 2012, 5:38pm UTC](https://discuss.elastic.co/t/removing-whitespace-around-a-delimiter-in-a-custom-anaylzer/6624/5 "2012-02-08T17:38:47Z")

</div>

The original data looks like this:  
I am a token, I'm a token too, Tokenize me,This is a token,Tokenize me

Your guidance is very much appreciated.

I tried this WordDelimiter filter as part of a custom analyzer, and  
all it did was tokenize based on whitespace. Is there more  
information on how to use the type\_table field? What tokenizer should  
a custom analyzer that specifies a filter use?

"filter":{"comma\_delimiter":{"type":"word\_delimited","type\_table":  
{",":"SUBWORD\_DELIM"}}}

I feel like the solution should be easier than we're making it.

On Feb 8, 8:26 am, Karussell [tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com) wrote:

> > That appears to do the opposite of what I need.
> 
> I think you can hack this worddelimiter thing a lot. E.g. overwriting  
> the comma char to be recognized as SUBWORD\_DELIM (see type\_table)
> 
> > I have a field in my index that looks like this
> 
> do you mean the field or the original data?
> 
> > Any thoughts on how I can force the regex engine to be greedier in its analysis?
> 
> No idea, I'm avoiding regex when and where I can 🙂
> 
> so I would do this via a custom WhitespaceTokenizer which overwrites  
> isTokenChar
> 
> Peter.

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [February 8, 2012, 5:55pm UTC](https://discuss.elastic.co/t/removing-whitespace-around-a-delimiter-in-a-custom-anaylzer/6624/6 "2012-02-08T17:55:39Z")

</div>

On Wed, 2012-02-08 at 09:38 -0800, Rick Thomas wrote:

> The original data looks like this:  
> I am a token, I'm a token too, Tokenize me,This is a token,Tokenize me

your original example works for me:

curl -XPUT '[http://127.0.0.1:9200/foo/?pretty=1](http://127.0.0.1:9200/foo/?pretty=1)' -d '  
{  
"settings" : {  
"analysis" : {  
"analyzer" : {  
"comma" : {  
"pattern" : "\s\*,\s\*",  
"type" : "pattern"  
}  
}  
}  
}  
}  
'

curl -XGET '[http://127.0.0.1:9200/foo/\_analyze?pretty=1&text=I+am+a+token%2C+I'm+a+token+too%2C+Tokenize+me%2CThis+is+a+token%2CTokenize+me&analyzer=comma'](http://127.0.0.1:9200/foo/_analyze?pretty=1&text=I+am+a+token%2C+I'm+a+token+too%2C+Tokenize+me%2CThis+is+a+token%2CTokenize+me&analyzer=comma')

# [Wed Feb 8 18:54:41 2012] Response:

# {

# "tokens" : [

# {

# "end\_offset" : 12,

# "position" : 1,

# "start\_offset" : 0,

# "type" : "word",

# "token" : "i am a token"

# },

# {

# "end\_offset" : 29,

# "position" : 2,

# "start\_offset" : 14,

# "type" : "word",

# "token" : "i'm a token too"

# },

# {

# "end\_offset" : 42,

# "position" : 3,

# "start\_offset" : 31,

# "type" : "word",

# "token" : "tokenize me"

# },

# {

# "end\_offset" : 58,

# "position" : 4,

# "start\_offset" : 43,

# "type" : "word",

# "token" : "this is a token"

# },

# {

# "end\_offset" : 70,

# "position" : 5,

# "start\_offset" : 59,

# "type" : "word",

# "token" : "tokenize me"

# }

# ]

# }

Perhaps you need to give a working example (as above) showing exactly  
what you are doing, the results you are getting and what is wrong with  
those results

clint

---

<div class="post-metadata">

**Author:** ![Rick\_Thomas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rick_thomas/32/2993_2.png) [@Rick\_Thomas](https://discuss.elastic.co/u/Rick_Thomas)\
**Post date:** [February 8, 2012, 8:41pm UTC](https://discuss.elastic.co/t/removing-whitespace-around-a-delimiter-in-a-custom-anaylzer/6624/7 "2012-02-08T20:41:17Z")

</div>

Everything seems to work fine with the basic analysis, but when you  
introduce faceting, you get extra whitespace around the tokens that  
you don't get when you call \_analyze. To use real world data:

item 1: category: "foo bar "  
item 2: category: "foo bar, ding bar "

This will create 3 distinct tokens: "foo bar", "foo bar ", "ding bar  
"

In reality, I need 2 distinct tokens "foo bar" "ding bar"

I've put together a gist to recreate the issue:

> <https://gist.github.com/rickthomasjr/1773423>

Any help getting rid of the whitespace around the tokens would be much  
appreciated.

---

<div class="post-metadata">

**Author:** ![Rick\_Thomas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rick_thomas/32/2993_2.png) [@Rick\_Thomas](https://discuss.elastic.co/u/Rick_Thomas)\
**Post date:** [February 9, 2012, 9:44pm UTC](https://discuss.elastic.co/t/removing-whitespace-around-a-delimiter-in-a-custom-anaylzer/6624/8 "2012-02-09T21:44:55Z")

</div>

Is there anything I can do to simplify recreating the problem for  
those who are more knowledgable?

On Feb 8, 3:41 pm, Rick Thomas [rtho...@igodigital.com](mailto:rtho...@igodigital.com) wrote:

> Everything seems to work fine with the basic analysis, but when you  
> introduce faceting, you get extrawhitespacearound the tokens that  
> you don't get when you call \_analyze. To use real world data:
> 
> item 1: category: "foo bar "  
> item 2: category: "foo bar, ding bar "
> 
> This will create 3 distinct tokens: "foo bar", "foo bar ", "ding bar  
> "
> 
> In reality, I need 2 distinct tokens "foo bar" "ding bar"
> 
> I've put together a gist to recreate the issue:
> 
> [Faceting with commas and extra spaces · GitHub](https://gist.github.com/1773423)
> 
> Any help getting rid of thewhitespacearound the tokens would be much  
> appreciated.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [February 12, 2012, 11:33am UTC](https://discuss.elastic.co/t/removing-whitespace-around-a-delimiter-in-a-custom-anaylzer/6624/9 "2012-02-12T11:33:26Z")

</div>

On my end, not a regex expert, so not sure why whitepsaces are not removed based on your regular expression. Require some playing to get it done properly. One thing we can do is add an analyzer token filter that can trim whitespaces, might make things simpler...

On Thursday, February 9, 2012 at 11:44 PM, Rick Thomas wrote:

> Is there anything I can do to simplify recreating the problem for  
> those who are more knowledgable?
> 
> On Feb 8, 3:41 pm, Rick Thomas \<[rtho...@igodigital.com](mailto:rtho...@igodigital.com) ([http://igodigital.com](http://igodigital.com))\> wrote:
> 
> > Everything seems to work fine with the basic analysis, but when you  
> > introduce faceting, you get extrawhitespacearound the tokens that  
> > you don't get when you call \_analyze. To use real world data:
> > 
> > item 1: category: "foo bar "  
> > item 2: category: "foo bar, ding bar "
> > 
> > This will create 3 distinct tokens: "foo bar", "foo bar ", "ding bar  
> > "
> > 
> > In reality, I need 2 distinct tokens "foo bar" "ding bar"
> > 
> > I've put together a gist to recreate the issue:
> > 
> > [Faceting with commas and extra spaces · GitHub](https://gist.github.com/1773423)
> > 
> > Any help getting rid of thewhitespacearound the tokens would be much  
> > appreciated.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [February 12, 2012, 11:42am UTC](https://discuss.elastic.co/t/removing-whitespace-around-a-delimiter-in-a-custom-anaylzer/6624/10 "2012-02-12T11:42:38Z")

</div>

I added an issue here: [Analysis: Add trim token filter that trims whitespaces · Issue #1693 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1693).

On Sunday, February 12, 2012 at 1:33 PM, Shay Banon wrote:

> On my end, not a regex expert, so not sure why whitepsaces are not removed based on your regular expression. Require some playing to get it done properly. One thing we can do is add an analyzer token filter that can trim whitespaces, might make things simpler...
> 
> On Thursday, February 9, 2012 at 11:44 PM, Rick Thomas wrote:
> 
> > Is there anything I can do to simplify recreating the problem for  
> > those who are more knowledgable?
> > 
> > On Feb 8, 3:41 pm, Rick Thomas \<[rtho...@igodigital.com](mailto:rtho...@igodigital.com) ([http://igodigital.com](http://igodigital.com))\> wrote:
> > 
> > > Everything seems to work fine with the basic analysis, but when you  
> > > introduce faceting, you get extrawhitespacearound the tokens that  
> > > you don't get when you call \_analyze. To use real world data:
> > > 
> > > item 1: category: "foo bar "  
> > > item 2: category: "foo bar, ding bar "
> > > 
> > > This will create 3 distinct tokens: "foo bar", "foo bar ", "ding bar  
> > > "
> > > 
> > > In reality, I need 2 distinct tokens "foo bar" "ding bar"
> > > 
> > > I've put together a gist to recreate the issue:
> > > 
> > > [Faceting with commas and extra spaces · GitHub](https://gist.github.com/1773423)
> > > 
> > > Any help getting rid of thewhitespacearound the tokens would be much  
> > > appreciated.

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [February 13, 2012, 10:05am UTC](https://discuss.elastic.co/t/removing-whitespace-around-a-delimiter-in-a-custom-anaylzer/6624/11 "2012-02-13T10:05:38Z")

</div>

> I've put together a gist to recreate the issue:
> 
> [Faceting with commas and extra spaces · GitHub](https://gist.github.com/1773423)
> 
> Any help getting rid of the whitespace around the tokens would be much  
> appreciated.

OK - so the comma analyzer is actually removing whitespace around the  
comma. The problem is that you have whitespace at the beginning or end  
of your strings, where no commas are involved - that's where the  
whitespace is coming from.

This works:

curl -XPUT '[http://127.0.0.1:9200/foo/?pretty=1](http://127.0.0.1:9200/foo/?pretty=1)' -d '  
{  
"settings" : {  
"analysis" : {  
"analyzer" : {  
"comma" : {  
"pattern" : "^\s+|\s\*,\s\*|\s+$",  
"type" : "pattern"  
}  
}  
}  
}  
}  
'

clint

---

<div class="post-metadata">

**Author:** ![Rick\_Thomas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rick_thomas/32/2993_2.png) [@Rick\_Thomas](https://discuss.elastic.co/u/Rick_Thomas)\
**Post date:** [February 13, 2012, 8:40pm UTC](https://discuss.elastic.co/t/removing-whitespace-around-a-delimiter-in-a-custom-anaylzer/6624/12 "2012-02-13T20:40:30Z")

</div>

Thanks so much for the assistance!

On Feb 13, 5:05 am, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:

> > I've put together a gist to recreate the issue:
> 
> > [Faceting with commas and extra spaces · GitHub](https://gist.github.com/1773423)
> 
> > Any help getting rid of thewhitespacearound the tokens would be much  
> > appreciated.
> 
> OK - so the comma analyzer is actually removingwhitespacearound the  
> comma. The problem is that you havewhitespaceat the beginning or end  
> of your strings, where no commas are involved - that's where thewhitespaceis coming from.
> 
> This works:
> 
> curl -XPUT '[http://127.0.0.1:9200/foo/?pretty=1](http://127.0.0.1:9200/foo/?pretty=1)' -d '  
> {  
> "settings" : {  
> "analysis" : {  
> "analyzer" : {  
> "comma" : {  
> "pattern" : "^\s+|\s\*,\s\*|\s+$",  
> "type" : "pattern"  
> }  
> }  
> }  
> }}
> 
> '
> 
> clint

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:39am UTC](https://discuss.elastic.co/t/removing-whitespace-around-a-delimiter-in-a-custom-anaylzer/6624/13 "2017-07-06T03:39:35Z")

</div>