# Get distinct values from "text field" without remapping

**URL:** <https://discuss.elastic.co/t/get-distinct-values-from-text-field-without-remapping/309694>\
**Category:** Elasticsearch\
**Created:** [July 14, 2022, 11:41pm UTC](https://discuss.elastic.co/t/get-distinct-values-from-text-field-without-remapping/309694 "2022-07-14T23:41:09Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![lwwalker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lwwalker/32/52139_2.png) [@lwwalker](https://discuss.elastic.co/u/lwwalker)\
**Post date:** [July 14, 2022, 11:41pm UTC](https://discuss.elastic.co/t/get-distinct-values-from-text-field-without-remapping/309694/1 "2022-07-14T23:41:09Z")

</div>

I'm querying ~350 TB of documents.

Re-indexing is not an option.

Performance, within reason, is not a concern.

my documents have a field `s3_filename` `{"type": "text"}`. It doesn't have any subfields.

Setting `fielddata: true` for `s3_filename` gives the wrong results (it returns unique "words" from the value, not unique values)

e.g. `s3_filename` has values similar to `ud20220711/long-file-name-20220711.json.gz`

I want the aggregates on the value (`ud20220711/long-file-name-20220711.json.gz`), however, I get instead aggregates of the words:

returns buckets of parts of the `s3_filename`. E.g, [`long`, `file`, `name`, `20220711`, `json`, `gz`] but no bucket for the value `ud20220711/long-file-name-20220711.json.gz`

I've tried simple aggs, and aggs with composites, but nothing works...

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 15, 2022, 12:08am UTC](https://discuss.elastic.co/t/get-distinct-values-from-text-field-without-remapping/309694/2 "2022-07-15T00:08:21Z")

</div>

Which version of Elasticsearch are you using? Creating a runtime field with the `keyword` type would help you, but this is only available from `7.12+` if I'm not wrong.

---

<div class="post-metadata">

**Author:** ![lwwalker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lwwalker/32/52139_2.png) [@lwwalker](https://discuss.elastic.co/u/lwwalker)\
**Post date:** [July 15, 2022, 4:43am UTC](https://discuss.elastic.co/t/get-distinct-values-from-text-field-without-remapping/309694/3 "2022-07-15T04:43:41Z")

</div>

Thank you.

I'm running 7.16.1 currently.

That looks promising, but it is still complains about the text field `-(`

```auto
$ cat s3_filenames.json
{
  "runtime_mappings": {
    "s3_fn": {
      "type": "keyword",
      "script": {
        "source": "emit(doc['s3_filename'].value)"
      }
    }
  },
  "aggs": {
    "filenames": {
      "terms": {
        "field": "s3_fn",
        "size": 10
      }
    }
  }
}

```

throws this

```auto
    "failed_shards": [
      {
        "shard": 0,
        "index": "cloud-logs-003965",
        "node": "JZGAzVVoQEmmCqJJEu8nTg",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "script_stack": [
            "org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:875)",
            "org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:112)",
            "org.elasticsearch.index.query.SearchExecutionContext.lambda$lookup$2(SearchExecutionContext.java:512)",
            "org.elasticsearch.search.lookup.SearchLookup.getForField(SearchLookup.java:109)",
            "org.elasticsearch.search.lookup.LeafDocLookup$2.run(LeafDocLookup.java:107)",
            "org.elasticsearch.search.lookup.LeafDocLookup$2.run(LeafDocLookup.java:104)",
            "java.base/java.security.AccessController.doPrivileged(AccessController.java:318)",
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:104)",
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:28)",
            "emit(doc['s3_filename'].value)",
            " ^---- HERE"
          ],
          "script": "emit(doc['s3_filename'].value)",
          "lang": "painless",
          "position": {
            "offset": 9,
            "start": 0,
            "end": 30
          },
          "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [s3_filename] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
          }
        }
      }
    ]

```

---

<div class="post-metadata">

**Author:** ![lwwalker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lwwalker/32/52139_2.png) [@lwwalker](https://discuss.elastic.co/u/lwwalker)\
**Post date:** [July 15, 2022, 4:53am UTC](https://discuss.elastic.co/t/get-distinct-values-from-text-field-without-remapping/309694/4 "2022-07-15T04:53:33Z")

</div>

If I add `fielddata: true` to s3\_filename, then emit(doc['s3\_filname']) returns a list of words instead of the value.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 15, 2022, 5:01am UTC](https://discuss.elastic.co/t/get-distinct-values-from-text-field-without-remapping/309694/5 "2022-07-15T05:01:09Z")

</div>

Try to define the runtime field [without a script](https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime-mapping-fields.html#runtime-fields-scriptless).

You can use just:

```auto
PUT your-index/_mapping
{
  "runtime": {
    "s3_filename": {
      "type": "keyword"
    }
  }
}

```

This will change the mapping of the field `s3_filename` to keyword on runtime (this can have some impact in the performance).

If for some reason you need to remove this runtime mapping, just use:

```auto
PUT your-index/_mapping
{
  "runtime": {
    "s3_filename": null
  }
}

```

---

<div class="post-metadata">

**Author:** ![lwwalker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lwwalker/32/52139_2.png) [@lwwalker](https://discuss.elastic.co/u/lwwalker)\
**Post date:** [July 15, 2022, 6:23am UTC](https://discuss.elastic.co/t/get-distinct-values-from-text-field-without-remapping/309694/6 "2022-07-15T06:23:01Z")

</div>

Thanks! That did it. It takes 18 times as long to run; BUT, it has valid output!!!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 12, 2022, 6:23am UTC](https://discuss.elastic.co/t/get-distinct-values-from-text-field-without-remapping/309694/7 "2022-08-12T06:23:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.