Kibana 6.2.4 won't start - illegal_argument_exception

[alexus]

while trying to start Kibana (6.2.4), I get following error:

kibana-xxx-custom | {"type":"error","@timestamp":"2018-06-14T14:54:58Z","tags":["warning","monitoring-ui","kibana-monitoring"],"pid":1,"level":"error","error":{"message":"[illegal_argument_exception] Fielddata is disabled on text fields by default. Set fielddata=true on [type] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.","name":"Error","stack":"[illegal_argument_exception] Fielddata is disabled on text fields by default. Set fielddata=true on [type] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead. :: {\"path\":\"/.kibana-xxx/_search\",\"query\":{\"ignore_unavailable\":true,\"filter_path\":\"aggregations.types.buckets\"},\"body\":\"{\\\"size\\\":0,\\\"query\\\":{\\\"terms\\\":{\\\"type\\\":[\\\"dashboard\\\",\\\"visualization\\\",\\\"search\\\",\\\"index-pattern\\\",\\\"graph-workspace\\\",\\\"timelion-sheet\\\"]}},\\\"aggs\\\":{\\\"types\\\":{\\\"terms\\\":{\\\"field\\\":\\\"type\\\",\\\"size\\\":6}}}}\",\"statusCode\":400,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"illegal_argument_exception\\\",\\\"reason\\\":\\\"Fielddata is disabled on text fields by default. Set fielddata=true on [type] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.\\\"}],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[{\\\"shard\\\":0,\\\"index\\\":\\\".kibana-xxx\\\",\\\"node\\\":\\\"NktsgTg8Q9-qlVC_LcU-bQ\\\",\\\"reason\\\":{\\\"type\\\":\\\"illegal_argument_exception\\\",\\\"reason\\\":\\\"Fielddata is disabled on text fields by default. Set fielddata=true on [type] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.\\\"}}]},\\\"status\\\":400}\"}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:295:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:254:7)\n at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)\n at IncomingMessage.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)\n at emitNone (events.js:91:20)\n at IncomingMessage.emit (events.js:185:7)\n at endReadableNT (_stream_readable.js:974:12)\n at _combinedTickCallback (internal/process/next_tick.js:80:11)\n at process._tickCallback (internal/process/next_tick.js:104:9)"},"message":"[illegal_argument_exception] Fielddata is disabled on text fields by default. Set fielddata=true on [type] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}

Please advise

[tsullivan (Tim Sullivan)]

After pasting the error into a JSON formatter, and pulling the message out of the error, it is more self-explanatory:

[illegal_argument_exception] Fielddata is disabled on text fields by default. Set fielddata=true on [type] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.

It's saying that a type field of the documents in the monitoring data can't be aggregated on because it's mapped as a text field. Field used as a key for a buckets aggregation should never be mapped as text because of the memory cost. Monitoring is built to map all any type field as keyword, which I can verify in a 6.2.4 build and look at the data mapping for my monitoring indices:

GET /.monitoring-*/_mapping

If the type fields in the monitoring data are mapped as text, then it means the Elasticsearch index template for the monitoring didn't install correctly, or you have a custom index template that is accidentally overriding system index templates.

I would first check your index templates and see if any of would accidentally apply to indices other than the ones you intended.

[alexus]

It's probably worth to mention that I'm running multiple Kibana instances (where each Kibana instance has KIBANA_INDEX configured to respective index).

... with that being said, I've noticed only one out of many Kibana indices in the list, maybe that's part of my issue as well?

# curl --silent --request GET $ELASTICSEARCH_URI/.monitoring-*/_mapping?pretty | grep .monitoring-kibana-*
  ".monitoring-kibana-6-2018.06.10" : {
  ".monitoring-kibana-6-2018.06.09" : {
  ".monitoring-kibana-6-2018.06.14" : {
  ".monitoring-kibana-6-2018.06.11" : {
  ".monitoring-kibana-6-2018.06.15" : {
  ".monitoring-kibana-6-2018.06.12" : {
  ".monitoring-kibana-6-2018.06.13" : {
# 

As far as type field being mapped as text goes:

# curl --silent --request GET $ELASTICSEARCH_URI/.monitoring-kibana-6-*/_mapping?pretty | grep -i text
# 

it sure doesn't look like it...

Please advise.

[tsullivan (Tim Sullivan)]

Hi, sorry for the delay in my followup.

The error logged shows the internal query that is failing, and it looks like a query against one of your .kibana indices rather than the .monitoring-kibana-* indices.

I think if you were to try this query, it would give the same message about fielddata:

POST /.kibana-xxx/_search
{
  "size": 0,
  "query": {
    "terms": {
      "type": [
        "dashboard",
        "visualization",
        "search",
        "index-pattern",
        "graph-workspace",
        "timelion-sheet"
      ]
    }
  },
  "aggs": {
    "types": {
      "terms": {
        "field": "type",
        "size": 6
      }
    }
  }
}

Take a look at your .kibana-* indices and see if they have custom mapping or there's a mapping collision happening that would make the type field.

When I check how the type field of my .kibana mapping is indexed, I can see that it is keyword:

# GET /.kibana/_mapping/doc/field/type
{
  ".kibana": {
    "mappings": {
      "doc": {
        "type": {
          "full_name": "type",
          "mapping": {
            "type": {
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}

[alexus]

Per your reply:

I always was thinking it has something to do with .kibana rather then .monitoring-kibana-*...

POST:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [type] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": ".kibana-6-xxx",
        "node": "oO14-kL6SuCerxsGAvTHmg",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [type] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
        }
      }
    ]
  },
  "status": 400
}

you're absolutely correct, error is there and as far mapping goes:

GET /_cat/templates/kibana*:

kibana_index_template:.kibana-yyy [.kibana-yyy] 0 
kibana_index_template:.kibana    [.kibana]    0 

... yet another weird part is only two kibana (template/indices) are shows up here and I have more than that, however I never set any custom mapping for .kibana-* indices (above is all out of the box).

GET:

{
  ".kibana-6-xxx": {
    "mappings": {
      "doc": {
        "type": {
          "full_name": "type",
          "mapping": {
            "type": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        }
      }
    }
  }
}

How does one address this issue? maybe reindex?

Please advise.