# Aggregations based on text fields instead of keyword fields **URL:** \ **Category:** Elasticsearch\ **Created:** [April 1, 2019, 12:35pm UTC](https://discuss.elastic.co/t/aggregations-based-on-text-fields-instead-of-keyword-fields/174781 "2019-04-01T12:35:25Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![David9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david9/32/42150_2.png) [@David9](https://discuss.elastic.co/u/David9)\ **Post date:** [April 1, 2019, 12:35pm UTC](https://discuss.elastic.co/t/aggregations-based-on-text-fields-instead-of-keyword-fields/174781/1 "2019-04-01T12:35:25Z")
Let's say we are analyzing posts to a sports forum, where the records contain a title, a body (text) field, and keyword fields, where the keywords might be the name of the sport that the post concerns. I know how to do aggregations around keywords, so that I could report on all the posts that have the keyword 'tennis'. But let's say that I want to do queries in the same manner but the term is inside the _text_ body field, not in a keyword. So e.g. I want to aggregate on every post that contains the word 'exercise' somewhere in the body field, just as though that was a keyword field even though it isn't. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [April 1, 2019, 12:44pm UTC](https://discuss.elastic.co/t/aggregations-based-on-text-fields-instead-of-keyword-fields/174781/2 "2019-04-01T12:44:14Z")
I suspect you want to take a look at the [significant\_text](https://www.elastic.co/guide/en/elasticsearch/reference/master/search-aggregations-bucket-significanttext-aggregation.html) aggregation ---
**Author:** ![David9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david9/32/42150_2.png) [@David9](https://discuss.elastic.co/u/David9)\ **Post date:** [April 1, 2019, 2:11pm UTC](https://discuss.elastic.co/t/aggregations-based-on-text-fields-instead-of-keyword-fields/174781/3 "2019-04-01T14:11:06Z")
Thank you! Will try it. ---
**Author:** ![David9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david9/32/42150_2.png) [@David9](https://discuss.elastic.co/u/David9)\ **Post date:** [April 1, 2019, 2:32pm UTC](https://discuss.elastic.co/t/aggregations-based-on-text-fields-instead-of-keyword-fields/174781/4 "2019-04-01T14:32:33Z")
This is a related question: would it be possible instead to do an aggregation that shows e.g. the top-20 twitter handles (any term that starts with '@') mentioned in the text field of all the records, i.e. show me who is being mentioned the most in a corpus of documents, where I also can use filters to define what that corpus is, e.g. last month's documents. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [April 1, 2019, 2:43pm UTC](https://discuss.elastic.co/t/aggregations-based-on-text-fields-instead-of-keyword-fields/174781/5 "2019-04-01T14:43:35Z")
> [@David9](#): > > would it be possible instead to do an aggregation that shows e.g. the top-20 twitter handles Yes. This is best achieved if you prepare the content first - aka "entity extraction". Twitter handles are easy to spot and can be extracted with a regular expression into a structured field. So given a doc like this: ``` { "text": "Elasticsearch was created by @kimchy"} ``` You would turn it into : ``` { "text": "Elasticsearch was created by @kimchy", "mentions": ["@kimchy"] } ``` You could then use the `terms` aggregation or `significant_terms` aggregation on the `mentions` field. The entity extraction could be done upstream of elasticsearch in logstash or whatever programming language you might use to create docs or inside elasticsearch using an ingest pipeline configured with a processor like [grok](https://www.elastic.co/guide/en/elasticsearch/reference/master/grok-processor.html). In each case a regular expression is typically spotting the entities of interest (in your case Twitter handles) ---
**Author:** ![David9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david9/32/42150_2.png) [@David9](https://discuss.elastic.co/u/David9)\ **Post date:** [April 1, 2019, 2:52pm UTC](https://discuss.elastic.co/t/aggregations-based-on-text-fields-instead-of-keyword-fields/174781/6 "2019-04-01T14:52:04Z")
Thanks, this is excellent. I've pored over the docs recently looking for a capability (processor? plugin?) that makes it straightforward to 1) extract terms and then 2) write them back to the document. For the second part: does that have to be coded explicitly by hand, or is there some construct that does that for me? ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [April 1, 2019, 3:03pm UTC](https://discuss.elastic.co/t/aggregations-based-on-text-fields-instead-of-keyword-fields/174781/7 "2019-04-01T15:03:54Z")
I do most of my doc preparation in Python so probably not best positioned to comment on logstash/ingest pipelines. [This tool](https://regex101.com/) is great for testing and generating regex code to do the parsing. To get more eyeballs I suggest opening another issue to discuss "how to extract Twitter handles using Logstash or ingest pipelines". ---
**Author:** ![David9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david9/32/42150_2.png) [@David9](https://discuss.elastic.co/u/David9)\ **Post date:** [April 1, 2019, 3:12pm UTC](https://discuss.elastic.co/t/aggregations-based-on-text-fields-instead-of-keyword-fields/174781/8 "2019-04-01T15:12:04Z")
Thanks, new post here: [How to extract terms into a new field while indexing](https://discuss.elastic.co/t/how-to-extract-terms-into-a-new-field-while-indexing/174805) ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [April 1, 2019, 3:16pm UTC](https://discuss.elastic.co/t/aggregations-based-on-text-fields-instead-of-keyword-fields/174781/9 "2019-04-01T15:16:18Z")
> [@David9](#): > > Thanks, new post here: Good post. My impression of Grok processor from the examples was that it was used for parsing log files with strictly ordered sequences of values in a line. Obviously free-text with phone numbers and twitter handles etc is more free-form so I'm keen to learn if it can be used for that. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 29, 2019, 3:16pm UTC](https://discuss.elastic.co/t/aggregations-based-on-text-fields-instead-of-keyword-fields/174781/10 "2019-04-29T15:16:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.