# Significant terms aggregation with non tokenized text

**URL:** <https://discuss.elastic.co/t/significant-terms-aggregation-with-non-tokenized-text/19973>\
**Category:** Elasticsearch\
**Created:** [September 26, 2014, 1:57am UTC](https://discuss.elastic.co/t/significant-terms-aggregation-with-non-tokenized-text/19973 "2014-09-26T01:57:10Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mike](https://avatars.discourse-cdn.com/v4/letter/m/58f4c7/32.png) [@Mike](https://discuss.elastic.co/u/Mike)\
**Post date:** [September 26, 2014, 1:57am UTC](https://discuss.elastic.co/t/significant-terms-aggregation-with-non-tokenized-text/19973/1 "2014-09-26T01:57:10Z")

</div>

I just tried using the significant terms aggregation on two text fields I  
have, and noticed that it doesn't seem to work on "non tokenized" fields.  
On my keyword tokenized field, I get 0 for the bg\_count, and it looks the  
same as a regular terms query with slighly different counts. When I used  
my regular tokenized query, I see the results differ, and I have bg\_counts.  
Why is this?

Here are my 2 fields and analyzer:

"properties":{  
"query" : {

```
    "type" : "multi_field",                                             
  
    "fields" : {                                                       
   
        "query" : { "type" : "string" },                       
  
        "queryUntouched" : { "type" : "string", "analyzer" : 

```

"myLowercaseAnalyzer" }  
}

```
}

```

}

"analyzer" : {

```
"myLowercaseAnalyzer" : {                                               
  
    "tokenizer" : "keyword",                                           
   
    "filter" : ["lowercase"]                                           
   
}                                                                       

```

}

When I send the significant terms aggregation against queryUntouched it  
looks the same as a regular terms agg, with bg\_count set to 0:

"aggs": {  
"pop": {  
"terms": {  
"field": "queryUntouched",  
"size": 3  
}  
},  
"sig": {  
"significant\_terms": {  
"field": "queryUntouched",  
"size": 3  
}  
}  
}

aggregations: {

- pop: {
  - buckets: [
    - {
      - key: yield curve
      - doc\_count: 102  
}

    - {
      - key: gdp
      - doc\_count: 70  
}  
]  
}

- sig: {
  - doc\_count: 62804
  - buckets: [
    - {
      - key: yield curve
      - doc\_count: 102
      - score: 7.200895615143776
      - bg\_count: 0  
}

    - {
      - key: gdp
      - doc\_count: 81
      - score: 4.540783692447051
      - bg\_count: 0  
}  
]  
}

When I use the tokenized field, I get results that I would expect:  
"aggs": {  
"pop": {  
"terms": {  
"field": "query",  
"size": 2  
}  
},  
"sig": {  
"significant\_terms": {  
"field": "query",  
"size": 2  
}  
}  
}

aggregations: {

- pop: {
  - buckets: [
    - {
      - key: bank
      - doc\_count: 1423  
}

    - {
      - key: of
      - doc\_count: 641  
}  
]  
}

- sig: {
  - doc\_count: 62804
  - buckets: [
    - {
      - key: bank
      - doc\_count: 1423
      - score: 0.03191767117787348
      - bg\_count: 25686  
}

    - {
      - key: id
      - doc\_count: 715
      - score: 0.017449718916743313
      - bg\_count: 12274  
}  
]  
}

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e7a41870-bb42-46f5-9161-dbeb6c847ad2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e7a41870-bb42-46f5-9161-dbeb6c847ad2%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood\_2](https://avatars.discourse-cdn.com/v4/letter/m/c77e96/32.png) [@Mark\_Harwood\_2](https://discuss.elastic.co/u/Mark_Harwood_2)\
**Post date:** [September 26, 2014, 9:48am UTC](https://discuss.elastic.co/t/significant-terms-aggregation-with-non-tokenized-text/19973/2 "2014-09-26T09:48:32Z")

</div>

Unlike the terms aggs which only accesses the content loaded into RAM (aka  
FieldData), the significant\_terms agg has to also go to disk to check the  
frequency of terms in the index for the background count. This different  
datasource means the naming conventions can sometimes differ. Can you try  
prefix the field name used by the significant terms with "query" e.g.  
"field":"query.queryUnTouched"?

On Friday, September 26, 2014 2:57:10 AM UTC+1, Mike wrote:

> I just tried using the significant terms aggregation on two text fields I  
> have, and noticed that it doesn't seem to work on "non tokenized" fields.  
> On my keyword tokenized field, I get 0 for the bg\_count, and it looks the  
> same as a regular terms query with slighly different counts. When I used  
> my regular tokenized query, I see the results differ, and I have bg\_counts.  
> Why is this?
> 
> Here are my 2 fields and analyzer:
> 
> "properties":{  
> "query" : {
> 
> ```
> "type" : "multi_field",                                           
>     
> "fields" : {                                                       
>    
> "query" : { "type" : "string" },                     
>     
> "queryUntouched" : { "type" : "string", "analyzer" : 
> 
> ```
> 
> "myLowercaseAnalyzer" }  
> }
> 
> ```
> }
> 
> ```
> 
> }
> 
> "analyzer" : {
> 
> ```
> "myLowercaseAnalyzer" : {                                             
>     
> "tokenizer" : "keyword",                                           
>    
> "filter" : ["lowercase"]                                           
>    
> }                                                                     
> 
> ```
> 
> }
> 
> When I send the significant terms aggregation against queryUntouched it  
> looks the same as a regular terms agg, with bg\_count set to 0:
> 
> "aggs": {  
> "pop": {  
> "terms": {  
> "field": "queryUntouched",  
> "size": 3  
> }  
> },  
> "sig": {  
> "significant\_terms": {  
> "field": "queryUntouched",  
> "size": 3  
> }  
> }  
> }
> 
> aggregations: {
> 
> - pop: {
> - buckets: [
> - {
> - key: yield curve
> - doc\_count: 102  
> }
> 
> - {
> - key: gdp
> - doc\_count: 70  
> }  
> ]  
> }
> 
> - sig: {
> - doc\_count: 62804
> - buckets: [
> - {
> - key: yield curve
> - doc\_count: 102
> - score: 7.200895615143776
> - bg\_count: 0  
> }
> 
> - {
> - key: gdp
> - doc\_count: 81
> - score: 4.540783692447051
> - bg\_count: 0  
> }  
> ]  
> }
> 
> When I use the tokenized field, I get results that I would expect:  
> "aggs": {  
> "pop": {  
> "terms": {  
> "field": "query",  
> "size": 2  
> }  
> },  
> "sig": {  
> "significant\_terms": {  
> "field": "query",  
> "size": 2  
> }  
> }  
> }
> 
> aggregations: {
> 
> - pop: {
> - buckets: [
> - {
> - key: bank
> - doc\_count: 1423  
> }
> 
> - {
> - key: of
> - doc\_count: 641  
> }  
> ]  
> }
> 
> - sig: {
> - doc\_count: 62804
> - buckets: [
> - {
> - key: bank
> - doc\_count: 1423
> - score: 0.03191767117787348
> - bg\_count: 25686  
> }
> 
> - {
> - key: id
> - doc\_count: 715
> - score: 0.017449718916743313
> - bg\_count: 12274  
> }  
> ]  
> }

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/39153e77-d916-4132-8987-a89a88f0b8a5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/39153e77-d916-4132-8987-a89a88f0b8a5%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:59am UTC](https://discuss.elastic.co/t/significant-terms-aggregation-with-non-tokenized-text/19973/3 "2017-07-06T00:59:43Z")

</div>