# Length Token Filter
**URL:** https://discuss.elastic.co/t/length-token-filter/8282
**Category:** Elasticsearch
**Created:** [July 2, 2012, 12:38pm UTC](https://discuss.elastic.co/t/length-token-filter/8282 "2012-07-02T12:38:23Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@windoz](https://discuss.elastic.co/u/windoz)
#### Post date: [July 2, 2012, 12:38pm UTC](https://discuss.elastic.co/t/length-token-filter/8282/1 "2012-07-02T12:38:23Z")
I'm new to Elasticsearch and want to know how I can use the length token
filter. I'm trying to limit my search to exclude two letter words.
---
### Author:  [@drewr](https://discuss.elastic.co/u/drewr)
#### Post date: [July 2, 2012, 2:15pm UTC](https://discuss.elastic.co/t/length-token-filter/8282/2 "2012-07-02T14:15:14Z")
windoz wrote:
> I'm new to Elasticsearch and want to know how I can use the length
> token filter. I'm trying to limit my search to exclude two letter
> words.
Can you give us an example of what you've tried based on the
documentation?
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
Note that there's a length token filter at the bottom of the sample
config:
```
myTokenFilter2 :
type : length
min : 0
max : 2000
```
-Drew
---
### Author:  [@windoz](https://discuss.elastic.co/u/windoz)
#### Post date: [July 3, 2012, 8:43am UTC](https://discuss.elastic.co/t/length-token-filter/8282/3 "2012-07-03T08:43:16Z")
I've been reading around and found that I can use a custom analyzer with
custom stop. But It is not working. I posted the analyzer (custom.txt
)below using curl - X POST --data "@custom.txt"
http//localhost:9200/sample/test/1 on Windows OS. Is the correct way of
using the analyzers.
Here is custom.txt
{
"analysis": {
"analyzer": {
"symphony\_fulltext" : {
"type": "custom",
"tokenizer" : "standard",
"filter": ["stop", "asciifolding", "snowball", "lowercase",
"custom\_synonyms", "custom\_stop"]
},
"symphony\_autocomplete" : {
"type": "custom",
"tokenizer" : "standard",
"filter": ["asciifolding", "lowercase"]
}
},
"filter" : {
"custom\_synonyms": {
"type": "synonym",
"ignore\_case": "true",
"synonyms": [
"i-pod, i pod =\> ipod",
"definately, definitly, definetly =\> definitely"
]
},
"custom\_stop": {
"type": "stop",
"stopwords": ["a", "an", "and", "are", "as", "at", "be", "but", "by",
"into", "is", "it", "of", "on", "or", "such", "that", "the", "their",
"there", "these", "they", "this", "to", "was", "will"]
}
}
}
}
On Monday, July 2, 2012 2:38:23 PM UTC+2, windoz wrote:
> I'm new to Elasticsearch and want to know how I can use the length token
> filter. I'm trying to limit my search to exclude two letter words.
---
### Author:  [@drewr](https://discuss.elastic.co/u/drewr)
#### Post date: [July 3, 2012, 2:20pm UTC](https://discuss.elastic.co/t/length-token-filter/8282/4 "2012-07-03T14:20:55Z")
windoz wrote:
> I've been reading around and found that I can use a custom analyzer with
> custom stop. But It is not working. I posted the analyzer (custom.txt
> )below using curl - X POST --data "@custom.txt"
> http//localhost:9200/sample/test/1 on Windows OS. Is the correct way of
> using the analyzers.
[...]
By sending that data to /sample/test/1, you're just indexing it as a
regular doc in ES. You need to store it as part of your index
settings. Try something like this:
curl -s -XPUT localhost:9200/test
-d @\<(curl -s [http://p.draines.com/13413250381814c87452d.txt](http://p.draines.com/13413250381814c87452d.txt))
Then you can check the settings with:
curl -s localhost:9200/test/\_settings?pretty=1
-Drew
---
### Author:  [@windoz](https://discuss.elastic.co/u/windoz)
#### Post date: [July 4, 2012, 10:37am UTC](https://discuss.elastic.co/t/length-token-filter/8282/5 "2012-07-04T10:37:03Z")
I tried what you said, but unfortunately when I do my search for the top
ten most used words in the documents I still get the stop words being
indexed. What could be the problem?
On Tuesday, July 3, 2012 4:20:55 PM UTC+2, Drew Raines wrote:
> windoz wrote:
>
> > I've been reading around and found that I can use a custom analyzer with
> > custom stop. But It is not working. I posted the analyzer (custom.txt
> > )below using curl - X POST --data "@custom.txt"
> > http//localhost:9200/sample/test/1 on Windows OS. Is the correct way
> > of
> > using the analyzers.
>
> [...]
>
> By sending that data to /sample/test/1, you're just indexing it as a
> regular doc in ES. You need to store it as part of your index
> settings. Try something like this:
>
> curl -s -XPUT localhost:9200/test \
> -d @\<(curl -s [http://p.draines.com/13413250381814c87452d.txt](http://p.draines.com/13413250381814c87452d.txt))
>
> Then you can check the settings with:
>
> curl -s localhost:9200/test/\_settings?pretty=1
>
> -Drew
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [July 5, 2012, 9:46pm UTC](https://discuss.elastic.co/t/length-token-filter/8282/6 "2012-07-05T21:46:09Z")
Are you correctly apply your analyzer as the mapping of your field?
Can you gist your mapping as well?
--
Ivan
On Wed, Jul 4, 2012 at 3:37 AM, windoz [victor.21.marisa@gmail.com](mailto:victor.21.marisa@gmail.com) wrote:
> I tried what you said, but unfortunately when I do my search for the top ten
> most used words in the documents I still get the stop words being indexed.
> What could be the problem?
>
> On Tuesday, July 3, 2012 4:20:55 PM UTC+2, Drew Raines wrote:
>
> > windoz wrote:
> >
> > > I've been reading around and found that I can use a custom analyzer with
> > > custom stop. But It is not working. I posted the analyzer (custom.txt
> > > )below using curl - X POST --data "@custom.txt"
> > > http//localhost:9200/sample/test/1 on Windows OS. Is the correct way
> > > of
> > > using the analyzers.
> >
> > [...]
> >
> > By sending that data to /sample/test/1, you're just indexing it as a
> > regular doc in ES. You need to store it as part of your index
> > settings. Try something like this:
> >
> > curl -s -XPUT localhost:9200/test
> > -d @\<(curl -s [http://p.draines.com/13413250381814c87452d.txt](http://p.draines.com/13413250381814c87452d.txt))
> >
> > Then you can check the settings with:
> >
> > curl -s localhost:9200/test/\_settings?pretty=1
> >
> > -Drew
---
### Author:  [@windoz](https://discuss.elastic.co/u/windoz)
#### Post date: [July 6, 2012, 8:32am UTC](https://discuss.elastic.co/t/length-token-filter/8282/7 "2012-07-06T08:32:49Z")
I'm now trying a new way, shown below, If i use a query to search the top
ten words in the message field of the docs index, I still get the words
[is, the, this,....] that i have included in the stop words list in my
custom filter. The search\_analyzer deals with the searching part and
index\_analyzer with the indexing part.
Here is the mapping, analyzers and filters.
{
"mappings" : {
"message" : {
"properties" : {
"title" : {
"type" : "string",
"search\_analyzer" : "str\_search\_analyzer",
"index\_analyzer" : "str\_index\_analyzer"
```
}
}
}
```
},
"settings" : {
"analysis" : {
"analyzer" : {
"str\_search\_analyzer" : {
"tokenizer" : "keyword",
"filter" : ["lowercase","custom\_stop"]
},
```
"str_index_analyzer" : {
"tokenizer" : "keyword",
"filter" : ["lowercase",]
}
},
"filter" :
```
"custom\_stop": {
"type": "stop",
"stopwords": ["a", "an", "and", "are", "as", "at", "be", "but", "by",
"into", "is", "it", "of", "on", "or", "such", "that", "the", "their",
"there", "these", "they", "this", "to", "was", "will","we"]
}
}
}
}
}
On Thursday, July 5, 2012 11:46:09 PM UTC+2, Ivan Brusic wrote:
> Are you correctly apply your analyzer as the mapping of your field?
> Can you gist your mapping as well?
>
> --
> Ivan
>
> On Wed, Jul 4, 2012 at 3:37 AM, windoz wrote:
>
> > I tried what you said, but unfortunately when I do my search for the top
> > ten
> > most used words in the documents I still get the stop words being
> > indexed.
> > What could be the problem?
> >
> > On Tuesday, July 3, 2012 4:20:55 PM UTC+2, Drew Raines wrote:
> >
> > > windoz wrote:
> > >
> > > > I've been reading around and found that I can use a custom analyzer
> > > > with
> > > > custom stop. But It is not working. I posted the analyzer (custom.txt
> > > > )below using curl - X POST --data "@custom.txt"
> > > > http//localhost:9200/sample/test/1 on Windows OS. Is the correct
> > > > way
> > > > of
> > > > using the analyzers.
> > >
> > > [...]
> > >
> > > By sending that data to /sample/test/1, you're just indexing it as a
> > > regular doc in ES. You need to store it as part of your index
> > > settings. Try something like this:
> > >
> > > curl -s -XPUT localhost:9200/test \
> > > -d @\<(curl -s [http://p.draines.com/13413250381814c87452d.txt](http://p.draines.com/13413250381814c87452d.txt))
> > >
> > > Then you can check the settings with:
> > >
> > > curl -s localhost:9200/test/\_settings?pretty=1
> > >
> > > -Drew
---
### Author:  [@drewr](https://discuss.elastic.co/u/drewr)
#### Post date: [July 6, 2012, 7:26pm UTC](https://discuss.elastic.co/t/length-token-filter/8282/8 "2012-07-06T19:26:50Z")
windoz wrote:
> I'm now trying a new way, shown below, If i use a query to search the top
> ten words in the message field of the docs index, I still get the words
> [is, the, this,....] that i have included in the stop words list in my
> custom filter.
Can you provide a script that reproduces what you're seeing and what
you would like it to do instead? Something that sets up your index,
indexes something, queries, and then tell us how it differs from what
you expected.
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
-Drew
---
### Author:  [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)
#### Post date: [July 8, 2012, 4:19pm UTC](https://discuss.elastic.co/t/length-token-filter/8282/9 "2012-07-08T16:19:36Z")
Hi windoz,
There is a couple of syntax errors in your example. A curly braket is
missing here:
"filter" :
"custom\_stop": {
And a filter is missing in str\_index\_analyzer definition:
```
"str_index_analyzer" : {
"tokenizer" : "keyword",
```
-
```
"filter" : ["lowercase",]*
}
```
I also don't think that "keyword" tokenizer is what you want in your case.
It emits content of the entire field as a single token, which doesn't allow
stop word filter to do its job unless your fields consist of single words.
I think, it might be better to use standard tokenizer instead. With these
changes, this is how your example might look
like: [https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/C9lp8oHrg7k · GitHub](https://gist.github.com/3071582)
Igor
On Friday, July 6, 2012 3:26:50 PM UTC-4, Drew Raines wrote:
> windoz wrote:
>
> > I'm now trying a new way, shown below, If i use a query to search the
> > top
> > ten words in the message field of the docs index, I still get the words
> > [is, the, this,....] that i have included in the stop words list in my
> > custom filter.
>
> Can you provide a script that reproduces what you're seeing and what
> you would like it to do instead? Something that sets up your index,
> indexes something, queries, and then tell us how it differs from what
> you expected.
>
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/help/)
>
> -Drew
---
### Author:  [@windoz](https://discuss.elastic.co/u/windoz)
#### Post date: [July 9, 2012, 8:25am UTC](https://discuss.elastic.co/t/length-token-filter/8282/10 "2012-07-09T08:25:37Z")
Thanks Motov !
Your code seems to be working fine so far.
On Sunday, July 8, 2012 6:19:36 PM UTC+2, Igor Motov wrote:
> Hi windoz,
>
> There is a couple of syntax errors in your example. A curly braket is
> missing here:
>
> "filter" :
> "custom\_stop": {
>
> And a filter is missing in str\_index\_analyzer definition:
>
> ```
> "str_index_analyzer" : {
> "tokenizer" : "keyword",
>
> ```
>
> -
>
> ```
> "filter" : ["lowercase",]*
> }
>
> ```
>
> I also don't think that "keyword" tokenizer is what you want in your case.
> It emits content of the entire field as a single token, which doesn't allow
> stop word filter to do its job unless your fields consist of single words.
> I think, it might be better to use standard tokenizer instead. With these
> changes, this is how your example might look like:
> [https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/C9lp8oHrg7k · GitHub](https://gist.github.com/3071582)
>
> Igor
>
> On Friday, July 6, 2012 3:26:50 PM UTC-4, Drew Raines wrote:
>
> > windoz wrote:
> >
> > > I'm now trying a new way, shown below, If i use a query to search the
> > > top
> > > ten words in the message field of the docs index, I still get the words
> > > [is, the, this,....] that i have included in the stop words list in
> > > my
> > > custom filter.
> >
> > Can you provide a script that reproduces what you're seeing and what
> > you would like it to do instead? Something that sets up your index,
> > indexes something, queries, and then tell us how it differs from what
> > you expected.
> >
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/help/)
> >
> > -Drew
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 3:21am UTC](https://discuss.elastic.co/t/length-token-filter/8282/11 "2017-07-06T03:21:01Z")