# Aggregating matched tokens

**URL:** <https://discuss.elastic.co/t/aggregating-matched-tokens/305458>\
**Category:** Elasticsearch\
**Created:** [May 24, 2022, 6:50am UTC](https://discuss.elastic.co/t/aggregating-matched-tokens/305458 "2022-05-24T06:50:58Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pyppe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pyppe/32/3531_2.png) [@Pyppe](https://discuss.elastic.co/u/Pyppe)\
**Post date:** [May 24, 2022, 6:50am UTC](https://discuss.elastic.co/t/aggregating-matched-tokens/305458/1 "2022-05-24T06:50:59Z")

</div>

Hi!

Is there anyway to aggregate the "matched" tokens only for a particular field? Let's say we have a field where we tokenize it using whitespaces, and we have documents such as:

- "THE PINK PANTHER"
- "THE PUNK BAND"

And if we create regexp query `P.NK` we'd like to aggregate the `PINK` and and `PUNK` only (without `THE` token rising on top). Is there a way to achieve this?

Thanks!

---

<div class="post-metadata">

**Author:** ![vincenbr](https://avatars.discourse-cdn.com/v4/letter/v/8edcca/32.png) [@vincenbr](https://discuss.elastic.co/u/vincenbr)\
**Post date:** [May 24, 2022, 9:54am UTC](https://discuss.elastic.co/t/aggregating-matched-tokens/305458/2 "2022-05-24T09:54:06Z")

</div>

Hi Pyppe,  
I'm not sure what you mean by _"matched" tokens_. At first sight, I don't see a way to get all possible tokens that matched your text field (as in an inverted index query), and simultaneously aggregate on them (aggregrate implies that you have a keyword or fielddata).  
Fielddata would allow to aggregate on single tokens, but the problem is then : how to filter which token matched, and keep only corresponding buckets.  
On the other hand (regexp approach) you could use a runtime field to extract the tokens that match your regexp in a keyword field, and run a terms aggregation on this field, which is pretty straightforward.

---

<div class="post-metadata">

**Author:** ![Pyppe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pyppe/32/3531_2.png) [@Pyppe](https://discuss.elastic.co/u/Pyppe)\
**Post date:** [May 24, 2022, 10:15am UTC](https://discuss.elastic.co/t/aggregating-matched-tokens/305458/3 "2022-05-24T10:15:53Z")

</div>

By matched token I mean if `"THE PINK PANTHER"` has tokens `[THE, PINK, PANTHER]`, and query being `P.NK`, the matched token would be `PINK`. That is, when utilizing a whitespace tokenizer.

> "On the other hand (regexp approach) you could use a runtime field to extract the tokens that match your regexp in a keyword field, and run a terms aggregation on this field, which is pretty straightforward."

Haven't ever done something like this. Any tips for relevant documentation about using this runtime field, and running terms aggregation on it?

---

<div class="post-metadata">

**Author:** ![vincenbr](https://avatars.discourse-cdn.com/v4/letter/v/8edcca/32.png) [@vincenbr](https://discuss.elastic.co/u/vincenbr)\
**Post date:** [May 25, 2022, 12:23am UTC](https://discuss.elastic.co/t/aggregating-matched-tokens/305458/4 "2022-05-25T00:23:26Z")

</div>

First, you can read this blog post as an introduction [Getting started with runtime fields, Elastic’s implementation of schema on read | Elastic Blog](https://www.elastic.co/blog/getting-started-with-elasticsearch-runtime-fields) . Especially the first part, where a runtime field is declared inside a \_search request. If the runtime field is of type `keyword` , then it can be used in a terms aggregation, as any `keyword` field from your mapping.

In the following example, I have a `author` field, possibly multi-valued, which can contain values like  
"THE PINK PANTHER"  
"PUNK BAND"  
"STEVE PINK"  
I have added `"fielddata": true` in the mapping for this text field, as an optimization. It is more efficient as it enables the script to get data from doc\_values, but you can also use data from the `_source` (for ref: [Map a runtime field | Elasticsearch Guide [8.2] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime-mapping-fields.html)).  
The following query should do (more or less) what you want:

```auto
GET myindex/_search
{
  "runtime_mappings": {
    "toto": {
      "type": "keyword",
      "script": {
        "source": """
        List authors = doc['authors'];
        for (String s : authors)
        {
          def m = /p.nk/.matcher(s);
          if (m.find())
          emit (m.group());
        }
        """
      }
    }
  },
  "aggs": {
    "nom": {
      "terms": {
        "field": "toto",
        "size": 10
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 22, 2022, 12:23am UTC](https://discuss.elastic.co/t/aggregating-matched-tokens/305458/5 "2022-06-22T00:23:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.