Assistance Required Resolving Index Mapping Error on Text / Keyword Field

[elasticTrouble]

Howdy Elastic Community,

I've got what I believe is a relatively simple issue that I need assistance resolving. Even though I believe I know the root cause of the issue, I'm asking for help because I haven't fully wrapped my head around index mappings and best-practices when it comes to modifying index mappings.

The issue I'm trying to resolve is related to attempting to aggregate on a text field which is returning the " Fielddata is disabled on text fields by default" error, the full error message is below. As far as I can tell this is affecting my visualizations / dashboards (Other areas may be affected that I'm unaware of).

The issue began when my indices rolled over from September to October and from what I can gather the index mapping changed. Prior to October all of my indices were imported via the snapshot / restore API during a migration from ES 5.3 to 6.8 so this October index is the first index created within ES 6.8.

"Fielddata is disabled on text fields by default. Set fielddata=true on [ChannelPartner] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."

From the research I've done, Resolving this issue should be as easy as updating the mapping for the "ChannelPartner" field in the Responses-* index from "text" to "keyword". Based on the index mapping I have the "ChannelPartner" field is mapped as both a "text", and "keyword" field and I should be able to use "ChannelPartner.keyword" in my visualizations for aggregation. However, that field does not appear to exist. I've posted truncated mappings for my September "Response" index (unaffected by issue) and my October "Response" index below.

For what it's worth, my index pattern within Kibana specifies all of the response indices as so "responses-*" and the individual naming pattern is "responses-YYYY.MM". Due to the inconsistant index mapping between months could this also be part of the issue?

GET /responses-2019.10/_mapping/ [Affected index]
{
  "responses-2019.10" : {
    "mappings" : {
      "doc" : {
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          },
          "@version" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "ApplicationTimeStamp" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "Brand" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "ChannelPartner" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "Creditlimit" : {
            "type" : "float"
          },
          "Financedamount" : {
            "type" : "float"
          },
          "Lastname" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "Lender" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
}

GET /responses-2019.09/_mapping

{
  "responses-2019.09" : {
    "mappings" : {
      "_default_" : {
        "_all" : {
          "enabled" : false
        },
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          },
          "@version" : {
            "type" : "text"
          },
          "ApplicationTimeStamp" : {
            "type" : "text"
          },
          "Brand" : {
            "type" : "keyword"
          },
          "ChannelPartner" : {
            "type" : "keyword"
          }
       },
      "APIResponseLog" : {
        "_all" : {
          "enabled" : false
        },
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          },
          "@version" : {
            "type" : "text"
          },
          "ApplicationTimeStamp" : {
            "type" : "text"
          },
          "Brand" : {
            "type" : "keyword"
          },
          "ChannelPartner" : {
            "type" : "keyword"
          }
          }
}

What would my best path forward for resolving this issue be? As far as I understand it I believe I have three options to choose from:

1 - Assuming the "ChannelPartner.keyword" field does exist and It's simply a matter of creating a new index pattern that encompass the updated index mapping while leaving the old mappings behind I could then update my visualizations to utilize the new "Channelpartner.keyword" field.

2 - I could re-index all of my "Responses" indices prior to October using the new updated mapping using the reindex-api. I assume when I create the new indices I'll be re-indexing into I'll want to create an alias for them IE- "responses-2019.09.updated" that way my existing index pattern "responses-*" will still match indices properly.

3 - Enable "fielddata=true" within the index mapping for the "ChannelPartner" field. Which I really would rather avoid doing.

Resources Referenced:

Trying to use Aggregate, "Fielddata is disabled on text fields by default. Set fielddata=true"

hello sir @dadoonet, thank you for the fast response! may I know how to run my aggregation on author_id.keyword? is it supposed to look like this? 'aggs' => [ 'authors' => [ 'terms' => [ 'field' => 'author_id.keyword', 'size' => 25, 'order' => [ '_term' => 'asc' ] ] ], ] If it is correct, I've started retrieving buckets! Again, thank you very much, appreciate it!

"Fielddata is disabled on text fields by default" error on keyword field

I'm getting this error in my log: Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [component] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead. at org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:670) ~[elasticsearch-6.5.0.jar:6.5.0] Which I don't unders…

Elasticsearch replaces string type with two new types text and keyword.

[elasticTrouble]

Just wanted to post an update to this thread with my particular solution in-case anyone else finds this thread in the future.

Long story short, I did end up re-creating all of my old indices that were originally created using ES 5.3 and imported into my 6.8 cluster using index mappings that did not have multiple document mapping types.

Once I created "updated" indices with mappings that conformed to ES 6.8 standards I used the reindex API to copy data from the old indices into the new indices using the code snippet below. The "script" portion converts all documents to _type doc.

POST _reindex
{
  "source": {
    "index": "OLD-INDEX"
  },
  "dest": {
    "index": "NEW-INDEX"
  },
  "script": {
    "source": "ctx._type = 'doc'",
    "lang": "painless"
  }
}

Once all of my original data was re-indexed into updated indices I deleted the corresponding original indices (after taking an index snapshot of course) and then updated the index pattern field list using "management -> index patterns -> $select_your_index -> refresh field list".

Finally, I updated my visualizations / saved searches using the updated "parameter.keyword" fields.