Install Kibana 7.16 fails with parse mapping

[Karel_Cech (kcech)]

Hi, we try to install Kibana 7.16.3 in our Openshift environment, but it fails with
"Failed to parse mapping [_doc]" :

{"type":"log","@timestamp":"2022-03-29T12:09:24+02:00","tags":["info","savedobjects-service"],"pid":7,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2022-03-29T12:09:25+02:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana] INIT -> CREATE_NEW_TARGET. took: 190ms."}
{"type":"log","@timestamp":"2022-03-29T12:09:25+02:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 193ms."}
{"type":"log","@timestamp":"2022-03-29T12:09:25+02:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana] Unexpected Elasticsearch ResponseError: statusCode: 400, method: PUT, url: /.kibana_7.16.3_001?wait_for_active_shards=all&timeout=60s error: [mapper_parsing_exception]: Failed to parse mapping [_doc]: unknown parameter [enabled] on mapper [params] of type [flattened],"}
{"type":"log","@timestamp":"2022-03-29T12:09:25+02:00","tags":["fatal","root"],"pid":7,"message":"Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Unexpected Elasticsearch ResponseError: statusCode: 400, method: PUT, url: /.kibana_7.16.3_001?wait_for_active_shards=all&timeout=60s error: [mapper_parsing_exception]: Failed to parse mapping [_doc]: unknown parameter [enabled] on mapper [params] of type [flattened],\n    at migrationStateActionMachine (/usr/share/kibana/src/core/server/saved_objects/migrationsv2/migrations_state_action_machine.js:164:13)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async Promise.all (index 0)\n    at SavedObjectsService.start (/usr/share/kibana/src/core/server/saved_objects/saved_objects_service.js:181:9)\n    at Server.start (/usr/share/kibana/src/core/server/server.js:330:31)\n    at Root.start (/usr/share/kibana/src/core/server/root/index.js:6...
{"type":"log","@timestamp":"2022-03-29T12:09:25+02:00","tags":["info","plugins-system","standard"],"pid":7,"message":"Stopping all plugins."}
{"type":"log","@timestamp":"2022-03-29T12:09:25+02:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":7,"message":"Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2022-03-29T12:09:25+02:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 104ms."}
{"type":"log","@timestamp":"2022-03-29T12:09:25+02:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 15ms."}
{"type":"log","@timestamp":"2022-03-29T12:09:25+02:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 4ms."}
{"type":"log","@timestamp":"2022-03-29T12:09:25+02:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 178ms."}
{"type":"log","@timestamp":"2022-03-29T12:09:25+02:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE. took: 36ms."}
{"type":"log","@timestamp":"2022-03-29T12:09:25+02:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] Migration completed after 531ms"}
{"type":"log","@timestamp":"2022-03-29T12:09:55+02:00","tags":["warning","plugins-system","standard"],"pid":7,"message":"\"eventLog\" plugin didn't stop in 30sec., move on to the next."}

 FATAL  Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Unexpected Elasticsearch ResponseError: statusCode: 400, method: PUT, url: /.kibana_7.16.3_001?wait_for_active_shards=all&timeout=60s error: [mapper_parsing_exception]: Failed to parse mapping [_doc]: unknown parameter [enabled] on mapper [params] of type [flattened],

ES version is also 7.16.3.
Prior to the install, we deleted all Kibana indices with
curl -XDELETE http://localhost:9200/.kibana*

At this point we are kind of stuck with the install..

Thanks for any suggestions!

[Dzmitry]

Hi @Karel_Cech,

Since you mentioned deleting all Kibana indices, are you in progress with Stack upgrade?

Kibana is failing due to Elasticsearch error so I would start check your ES health and configuration.

Regards, Dima

[Karel_Cech (kcech)]

Hi @Dzmitry ,

thanks for the reply!

Actually, our aim was to update Kibana from 7.10, since ES is on 7.16.3.
Kibana 7.10 and ES 7.16.3 were working together just ok for about 2 months now.

Recently we saw that the versions are not the same, so we wanted to level Kibana to the same version of ES. This update didn't went actually well and we saw a similar mapper error on the same .kibana_7.16.3_001 index (mapper [cases.title] cannot be changed from type [keyword] to [text],)

That's why we decided to start over and delete all KIbana indices

Actually, the cluster health is ok:

GET http://localhost:9200/_cluster/health
{
    "cluster_name": "my_cluster",
    "status": "green",
    "timed_out": false,
    "number_of_nodes": 1,
    "number_of_data_nodes": 1,
    "active_primary_shards": 63,
    "active_shards": 63,
    "relocating_shards": 0,
    "initializing_shards": 0,
    "unassigned_shards": 0,
    "delayed_unassigned_shards": 0,
    "number_of_pending_tasks": 0,
    "number_of_in_flight_fetch": 0,
    "task_max_waiting_in_queue_millis": 0,
    "active_shards_percent_as_number": 100.0
}

[afharo (Alejandro Fernandez Haro)]

Hi @Karel_Cech,

The 2 mapping errors that you shared make me wonder: is there anything else setting the mappings for .kibana while Kibana is starting?

Either another instance of Kibana still in a previous version or an index template that affects .kibana or *?

AFAIK, the index .kibana_7.16.3_001 should be created with the correct mapping, but the errors that you shared point out to either another source setting the mappings before Kibana's upgrade logic kicks in or even worse: writing to the index before the mappings are set.

If deleting .kibana* is OK on your end, would you mind stopping Kibana, deleting the indices, and starting it again?

[Karel_Cech (kcech)]

Hello Alejandro and Dima,

I finally discovered that there was a legacy template called "tenant_template" that matched ".kibana_7*_*", among others (goes up to v9 ) We don't use SG anymore, and actually I kind of never used it, hence I did not know about that template.

So, when Kibana created the .kibana_7.16.3_001, the SG template was applied. Search Guard obviously did not know about all the future changes to the internal Kibana indices - I did a diff between the resulting mappings and there are quite a lot of differences indeed.

Thanks guys!