Issues with custom analyzer when running aggregate queries


(Jack) #1

Hello,

We are using Elasticsearch v5.3, the issue we are seeing is that the aggregates on a term are not working on a custom defined analyzer (keyword tokenizer).

We have created a custom analyzer:
{
"settings": {
"analysis" : {
"analyzer" : {
"custom_keyword_analyzer": {
"tokenizer": "keyword"
}
}
}
}
}

Which we are applying to a field like so:
"properties": {
"portfolio": {
"type": "text",
"analyzer": "custom_keyword_analyzer"
}
}

The issue we are running into is that when searching using that tokenizer it does not return any aggregated data, the buckets in the aggregate response are empty. Here is an example search query with an aggregate using this tokenized field:
"aggs": {
"days_total_price": {
"sum": {
"field":"cost"
}
},
"grouped_by_portfolio":{
"terms": {
"field":"portfolio"
}
}
}

However, if we create our mapping like so (instead of using the custom analyzer):
"properties": {
"portfolio": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
}
}

We are able to do a search using .raw:
"aggs": {
"days_total_price": {
"sum": {
"field":"cost"
}
},
"grouped_by_portfolio":{
"terms": {
"field":"portfolio.raw"
}
}
}

This search returns the aggregates in the 'buckets' as expected.

Based on the documentation and research (Help understanding keyword vs not_analyzed, https://www.elastic.co/guide/en/elasticsearch/reference/5.3/analysis-keyword-tokenizer.html) these two mappings should behave the same. What have we done wrong? Thank you!


(Thiago Souza) #2

A field of type "text" with a "keyword" analyzer is different from a field of type "keyword". I understand that the names might lead to confusion, but the fact is that you can not aggregate on a field of type "text" (regardless of it's analyzer)

There is one feature that might be helpful to you which allows using a field of type "keyword" while still doing processing the token which is called normalizer. Maybe this is helps you.

Cheers


(Hamza Karachiwala) #3

Thank you very much for the prompt response! This solution worked for us!


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.