# Unique tokenfilter issues?

**URL:** <https://discuss.elastic.co/t/unique-tokenfilter-issues/12554>\
**Category:** Elasticsearch\
**Created:** [June 26, 2013, 5:36pm UTC](https://discuss.elastic.co/t/unique-tokenfilter-issues/12554 "2013-06-26T17:36:44Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![mtthrok](https://avatars.discourse-cdn.com/v4/letter/m/8c91f0/32.png) [@mtthrok](https://discuss.elastic.co/u/mtthrok)\
**Post date:** [June 26, 2013, 5:36pm UTC](https://discuss.elastic.co/t/unique-tokenfilter-issues/12554/1 "2013-06-26T17:36:44Z")

</div>

Hi,

I've been trying to use the unique tokenfilter in my analyzer, but it seems  
to continue using duplicate tokens in scoring.

Here's a simple example of the problem:  
curl -XPUT '[http://localhost:9200/people\_search2/?pretty=1](http://localhost:9200/people_search2/?pretty=1)' -d '  
{  
"mappings" : {  
"person" : {  
"properties" : {  
"\_id" : {  
"type" : "integer",  
"index" : "no"  
},  
"names" : {  
"type" : "string",  
"store":true,  
"omit\_norms":true,  
"analyzer":"partial\_name",  
"index\_analyzer":"partial\_name"  
}  
}  
}  
},  
"settings" : {  
"analysis" : {  
"filter" : {  
"name\_ngrams" : {  
"side" : "front",  
"max\_gram" : 25,  
"min\_gram" : 2,  
"type" : "edgeNGram"  
},  
"unique\_token\_filter": {  
"type":"unique",  
"only\_on\_same\_position":false  
}  
},  
"analyzer" : {  
"partial\_name" : {  
"filter" : [  
"standard",  
"lowercase",  
"asciifolding",  
"name\_ngrams",  
"unique\_token\_filter"  
],  
"type" : "custom",  
"tokenizer" : "standard"  
}  
}  
}  
}  
}  
'

Using the \_analysis API, I was able to verify that the partial\_name  
analyzer only returns unique tokens. However, if I load in this test data:  
curl -XPOST  
'[http://ec2-54-224-89-81.compute-1.amazonaws.com:9200/\_bulk?pretty=1](http://ec2-54-224-89-81.compute-1.amazonaws.com:9200/_bulk?pretty=1)' -d '  
{"index" : {"\_index" : "people\_search2", "\_type" : "person", "\_id" : 1}}  
{"\_id" : 1, "names":["Tim","Cook","Apple", "Google"]}  
{"index" : {"\_index" : "people\_search2", "\_type" : "person", "\_id" : 2}}  
{"\_id" : 2,"names":["Tim","Cook","Tim Cook Consulting","Random Co"]}  
{"index" : {"\_index" : "people\_search2", "\_type" : "person", "\_id" : 3}}  
{"\_id" : 3, "names":["John","Smith","Standard","ABC inc"]}  
'

And then issue this query:  
curl -XPOST  
'[http://ec2-54-224-89-81.compute-1.amazonaws.com:9200/people\_search2/person/\_search?search\_type=dfs\_query\_then\_fetch](http://ec2-54-224-89-81.compute-1.amazonaws.com:9200/people_search2/person/_search?search_type=dfs_query_then_fetch)'  
-d '  
{  
"explain": true,  
"query": {  
"match": {  
"names": {  
"query": "Tim Cook"  
}  
}  
}  
}  
'

It will rank the doc with \_id=2 higher because it has Tim Cook twice in the  
document (explain shows a termFreq=2). Why is this happening?

Thanks!

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\
**Post date:** [June 26, 2013, 5:57pm UTC](https://discuss.elastic.co/t/unique-tokenfilter-issues/12554/2 "2013-06-26T17:57:20Z")

</div>

Why not simply omit term frequencies for that field?

What I think might be happening is that the analyzer is applied for each  
instance of the names field, so you will have two names fields with "tim".  
You are storing the field, so it is easy to verify.

[http://ec2-54-224-89-81.compute-1.amazonaws.com:9200/people\_search2/person/\_search/2?fields=names](http://ec2-54-224-89-81.compute-1.amazonaws.com:9200/people_search2/person/_search/2?fields=names)

--  
Ivan

On Wed, Jun 26, 2013 at 10:36 AM, [mtthrok@gmail.com](mailto:mtthrok@gmail.com) wrote:

> Hi,
> 
> I've been trying to use the unique tokenfilter in my analyzer, but it  
> seems to continue using duplicate tokens in scoring.
> 
> Here's a simple example of the problem:  
> curl -XPUT '[http://localhost:9200/people\_search2/?pretty=1](http://localhost:9200/people_search2/?pretty=1)' -d '  
> {  
> "mappings" : {  
> "person" : {  
> "properties" : {  
> "\_id" : {  
> "type" : "integer",  
> "index" : "no"  
> },  
> "names" : {  
> "type" : "string",  
> "store":true,  
> "omit\_norms":true,  
> "analyzer":"partial\_name",  
> "index\_analyzer":"partial\_name"  
> }  
> }  
> }  
> },  
> "settings" : {  
> "analysis" : {  
> "filter" : {  
> "name\_ngrams" : {  
> "side" : "front",  
> "max\_gram" : 25,  
> "min\_gram" : 2,  
> "type" : "edgeNGram"  
> },  
> "unique\_token\_filter": {  
> "type":"unique",  
> "only\_on\_same\_position":false  
> }  
> },  
> "analyzer" : {  
> "partial\_name" : {  
> "filter" : [  
> "standard",  
> "lowercase",  
> "asciifolding",  
> "name\_ngrams",  
> "unique\_token\_filter"  
> ],  
> "type" : "custom",  
> "tokenizer" : "standard"  
> }  
> }  
> }  
> }  
> }  
> '
> 
> Using the \_analysis API, I was able to verify that the partial\_name  
> analyzer only returns unique tokens. However, if I load in this test data:  
> curl -XPOST '  
> [http://ec2-54-224-89-81.compute-1.amazonaws.com:9200/\_bulk?pretty=1](http://ec2-54-224-89-81.compute-1.amazonaws.com:9200/_bulk?pretty=1)' -d '  
> {"index" : {"\_index" : "people\_search2", "\_type" : "person", "\_id" : 1}}  
> {"\_id" : 1, "names":["Tim","Cook","Apple", "Google"]}  
> {"index" : {"\_index" : "people\_search2", "\_type" : "person", "\_id" : 2}}  
> {"\_id" : 2,"names":["Tim","Cook","Tim Cook Consulting","Random Co"]}  
> {"index" : {"\_index" : "people\_search2", "\_type" : "person", "\_id" : 3}}  
> {"\_id" : 3, "names":["John","Smith","Standard","ABC inc"]}  
> '
> 
> And then issue this query:  
> curl -XPOST '  
> [http://ec2-54-224-89-81.compute-1.amazonaws.com:9200/people\_search2/person/\_search?search\_type=dfs\_query\_then\_fetch](http://ec2-54-224-89-81.compute-1.amazonaws.com:9200/people_search2/person/_search?search_type=dfs_query_then_fetch)'  
> -d '  
> {  
> "explain": true,  
> "query": {  
> "match": {  
> "names": {  
> "query": "Tim Cook"  
> }  
> }  
> }  
> }  
> '
> 
> It will rank the doc with \_id=2 higher because it has Tim Cook twice in  
> the document (explain shows a termFreq=2). Why is this happening?
> 
> Thanks!
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:29am UTC](https://discuss.elastic.co/t/unique-tokenfilter-issues/12554/3 "2017-07-06T02:29:20Z")

</div>