Kibana 6.2.4 won't start - illegal_argument_exception

while trying to start Kibana (6.2.4), I get following error:

kibana-xxx-custom | {"type":"error","@timestamp":"2018-06-14T14:54:58Z","tags":["warning","monitoring-ui","kibana-monitoring"],"pid":1,"level":"error","error":{"message":"[illegal_argument_exception] Fielddata is disabled on text fields by default. Set fielddata=true on [type] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.","name":"Error","stack":"[illegal_argument_exception] Fielddata is disabled on text fields by default. Set fielddata=true on [type] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead. :: {\"path\":\"/.kibana-xxx/_search\",\"query\":{\"ignore_unavailable\":true,\"filter_path\":\"aggregations.types.buckets\"},\"body\":\"{\\\"size\\\":0,\\\"query\\\":{\\\"terms\\\":{\\\"type\\\":[\\\"dashboard\\\",\\\"visualization\\\",\\\"search\\\",\\\"index-pattern\\\",\\\"graph-workspace\\\",\\\"timelion-sheet\\\"]}},\\\"aggs\\\":{\\\"types\\\":{\\\"terms\\\":{\\\"field\\\":\\\"type\\\",\\\"size\\\":6}}}}\",\"statusCode\":400,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"illegal_argument_exception\\\",\\\"reason\\\":\\\"Fielddata is disabled on text fields by default. Set fielddata=true on [type] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.\\\"}],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[{\\\"shard\\\":0,\\\"index\\\":\\\".kibana-xxx\\\",\\\"node\\\":\\\"NktsgTg8Q9-qlVC_LcU-bQ\\\",\\\"reason\\\":{\\\"type\\\":\\\"illegal_argument_exception\\\",\\\"reason\\\":\\\"Fielddata is disabled on text fields by default. Set fielddata=true on [type] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.\\\"}}]},\\\"status\\\":400}\"}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:295:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:254:7)\n at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)\n at IncomingMessage.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)\n at emitNone (events.js:91:20)\n at IncomingMessage.emit (events.js:185:7)\n at endReadableNT (_stream_readable.js:974:12)\n at _combinedTickCallback (internal/process/next_tick.js:80:11)\n at process._tickCallback (internal/process/next_tick.js:104:9)"},"message":"[illegal_argument_exception] Fielddata is disabled on text fields by default. Set fielddata=true on [type] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}

Please advise

After pasting the error into a JSON formatter, and pulling the message out of the error, it is more self-explanatory:

[illegal_argument_exception] Fielddata is disabled on text fields by default. Set fielddata=true on [type] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.

It's saying that a type field of the documents in the monitoring data can't be aggregated on because it's mapped as a text field. Field used as a key for a buckets aggregation should never be mapped as text because of the memory cost. Monitoring is built to map all any type field as keyword, which I can verify in a 6.2.4 build and look at the data mapping for my monitoring indices:

GET /.monitoring-*/_mapping

If the type fields in the monitoring data are mapped as text, then it means the Elasticsearch index template for the monitoring didn't install correctly, or you have a custom index template that is accidentally overriding system index templates.

I would first check your index templates and see if any of would accidentally apply to indices other than the ones you intended.

It's probably worth to mention that I'm running multiple Kibana instances (where each Kibana instance has KIBANA_INDEX configured to respective index).

... with that being said, I've noticed only one out of many Kibana indices in the list, maybe that's part of my issue as well?

# curl --silent --request GET $ELASTICSEARCH_URI/.monitoring-*/_mapping?pretty | grep .monitoring-kibana-*
  ".monitoring-kibana-6-2018.06.10" : {
  ".monitoring-kibana-6-2018.06.09" : {
  ".monitoring-kibana-6-2018.06.14" : {
  ".monitoring-kibana-6-2018.06.11" : {
  ".monitoring-kibana-6-2018.06.15" : {
  ".monitoring-kibana-6-2018.06.12" : {
  ".monitoring-kibana-6-2018.06.13" : {
# 

As far as type field being mapped as text goes:

# curl --silent --request GET $ELASTICSEARCH_URI/.monitoring-kibana-6-*/_mapping?pretty | grep -i text
# 

it sure doesn't look like it...

Please advise.

Hi, sorry for the delay in my followup.

The error logged shows the internal query that is failing, and it looks like a query against one of your .kibana indices rather than the .monitoring-kibana-* indices.

I think if you were to try this query, it would give the same message about fielddata:

POST /.kibana-xxx/_search
{
  "size": 0,
  "query": {
    "terms": {
      "type": [
        "dashboard",
        "visualization",
        "search",
        "index-pattern",
        "graph-workspace",
        "timelion-sheet"
      ]
    }
  },
  "aggs": {
    "types": {
      "terms": {
        "field": "type",
        "size": 6
      }
    }
  }
}

Take a look at your .kibana-* indices and see if they have custom mapping or there's a mapping collision happening that would make the type field.

When I check how the type field of my .kibana mapping is indexed, I can see that it is keyword:

# GET /.kibana/_mapping/doc/field/type
{
  ".kibana": {
    "mappings": {
      "doc": {
        "type": {
          "full_name": "type",
          "mapping": {
            "type": {
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}

Per your reply:

I always was thinking it has something to do with .kibana rather then .monitoring-kibana-*...

POST:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [type] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": ".kibana-6-xxx",
        "node": "oO14-kL6SuCerxsGAvTHmg",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [type] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
        }
      }
    ]
  },
  "status": 400
}

you're absolutely correct, error is there and as far mapping goes:

GET /_cat/templates/kibana*:

kibana_index_template:.kibana-yyy [.kibana-yyy] 0 
kibana_index_template:.kibana    [.kibana]    0 

... yet another weird part is only two kibana (template/indices) are shows up here and I have more than that, however I never set any custom mapping for .kibana-* indices (above is all out of the box).

GET:

{
  ".kibana-6-xxx": {
    "mappings": {
      "doc": {
        "type": {
          "full_name": "type",
          "mapping": {
            "type": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        }
      }
    }
  }
}

How does one address this issue? maybe reindex?

Please advise.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.