# Doing a significant text aggregation with a custom analyzer
**URL:** https://discuss.elastic.co/t/doing-a-significant-text-aggregation-with-a-custom-analyzer/313059
**Category:** Elasticsearch
**Created:** [August 27, 2022, 9:23am UTC](https://discuss.elastic.co/t/doing-a-significant-text-aggregation-with-a-custom-analyzer/313059 "2022-08-27T09:23:58Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@Ivo\_Tavares](https://discuss.elastic.co/u/Ivo_Tavares)
#### Post date: [August 27, 2022, 9:23am UTC](https://discuss.elastic.co/t/doing-a-significant-text-aggregation-with-a-custom-analyzer/313059/1 "2022-08-27T09:23:58Z")
I have an ES index with a mapping like this:
```auto
{'texts-temp': {'aliases': {},
'mappings': {'properties': {'data': {'properties':
(...)
'user_text': {'type': 'text',
'fields': {'word_tokenized': {'type': 'text',
'analyzer': 'text_analyzer'}}},
(...)
},
'settings': {'index': {'number_of_shards': '1',
'provided_name': 'texts-temp',
'creation_date': (...),
'analysis': {'filter': {'shingen_filter': {'max_shingle_size': '3',
'min_shingle_size': '2',
'type': 'shingle'}},
'analyzer': {'text_analyzer': {'filter': ['lowercase',
'stop',
'asciifolding',
'apostrophe',
'stemmer',
'shingen_filter'],
'type': 'custom',
'tokenizer': 'standard'}}},
```
When I do the following search:
```auto
search = Search(index = 'texts-temp')
q = Q("terms", data=list_urls_text)
search = search.query(q)
search = search.extra(track_total_hits=True)
reply = search.execute()
[x.data.user_text.word_tokenized for x in reply]
```
I get an empty list... However, if I look at the elements `x.data.user_text`, then I get the texts, but not tokenized.
What am I doing wrong that the index doesn't have the field `data.user_text.word_tokenized`?
Is this why my significant text aggregation returns empty?
```auto
{'query':{terms:{'data': list_urls_text}
'aggs':{'sample':{'sampler':{'shard_size':200},
'aggs':{'keywords':{'significant_tex':{'field': 'data.user_text.word_tokenized', 'size'=10, 'filter_duplicate_text':True}}...}
```
---
### Author:  [@Mark\_Harwood1](https://discuss.elastic.co/u/Mark_Harwood1)
#### Post date: [August 27, 2022, 6:19pm UTC](https://discuss.elastic.co/t/doing-a-significant-text-aggregation-with-a-custom-analyzer/313059/2 "2022-08-27T18:19:13Z")
> [@Ivo\_Tavares](#):
>
> What am I doing wrong that the index doesn't have the field `data.user_text.word_tokenized`?
It looks right. I suspect that field setting will help determine the correct analyzer but the aggregation doesn’t know that it should then try use that analyzer on the contents of the ‘data.user\_text’ field in the json. To give a helping hand try set the [source\_fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-significanttext-aggregation.html#_dealing_with_source_and_index_mappings) parameter to ‘data.user\_text’.
I’m away from computer so can’t confirm but I suspect this will work.
---
### Author:  [@Ivo\_Tavares](https://discuss.elastic.co/u/Ivo_Tavares)
#### Post date: [August 29, 2022, 8:34am UTC](https://discuss.elastic.co/t/doing-a-significant-text-aggregation-with-a-custom-analyzer/313059/3 "2022-08-29T08:34:39Z")
Hi Mark,
Thanks for the reply. Unfortunatly, your proposed solution didn't work. I still get the same reply:
```auto
{'sample': {'doc_count': 200,
'wordcloud': {'doc_count': 200, 'bg_count': 8357411, 'buckets': []}}}
```
---
### Author:  [@Mark\_Harwood1](https://discuss.elastic.co/u/Mark_Harwood1)
#### Post date: [August 29, 2022, 9:18am UTC](https://discuss.elastic.co/t/doing-a-significant-text-aggregation-with-a-custom-analyzer/313059/4 "2022-08-29T09:18:08Z")
Hmm. Can you share the JSON for the search request?
---
### Author:  [@Ivo\_Tavares](https://discuss.elastic.co/u/Ivo_Tavares)
#### Post date: [August 30, 2022, 10:49am UTC](https://discuss.elastic.co/t/doing-a-significant-text-aggregation-with-a-custom-analyzer/313059/5 "2022-08-30T10:49:27Z")
Hi Mark, sorry for the late reply. I was under some time constraint, and since I couldn't use it, I decided to do it locally, using a tfidf scoring system. It solved my problem... Maybe in the future I'll return to this. Thanks for the help, either way. 😉
---
### Author:  [@Mark\_Harwood1](https://discuss.elastic.co/u/Mark_Harwood1)
#### Post date: [August 31, 2022, 7:09am UTC](https://discuss.elastic.co/t/doing-a-significant-text-aggregation-with-a-custom-analyzer/313059/6 "2022-08-31T07:09:51Z")
Thanks for the update. Would be good to find out what went wrong if you return to this.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 28, 2022, 7:10am UTC](https://discuss.elastic.co/t/doing-a-significant-text-aggregation-with-a-custom-analyzer/313059/7 "2022-09-28T07:10:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.