Fail to start Kibana 7.16.2

Elastic Stack Kibana
1

I have Elasticsearch and kibana running on version 7.9.2 for a very long time without issues. Lately, when I started to deploy Elasticsearch and kibana 7.16.2, I see the 403 error below:


{"type":"log","@timestamp":"2022-01-07T00:21:30+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2022-01-07T00:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 34ms."}
{"type":"log","@timestamp":"2022-01-07T00:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 4ms."}
{"type":"log","@timestamp":"2022-01-07T00:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 11ms."}
{"type":"log","@timestamp":"2022-01-07T00:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 4ms."}
{"type":"log","@timestamp":"2022-01-07T00:21:31+00:00","tags":["error","savedobjects-service"],"pid":1,"message":"[.kibana] Action failed with '<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>403 Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p>You don't have permission to access this resource.</p>\n</body></html>\n'. Retrying attempt 1 in 2 seconds."}
{"type":"log","@timestamp":"2022-01-07T00:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS. took: 85ms."}
{"type":"log","@timestamp":"2022-01-07T00:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 149ms."}
{"type":"log","@timestamp":"2022-01-07T00:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 7ms."}
{"type":"log","@timestamp":"2022-01-07T00:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 25ms."}
{"type":"log","@timestamp":"2022-01-07T00:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 5ms."}
{"type":"log","@timestamp":"2022-01-07T00:21:31+00:00","tags":["error","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] Action failed with '<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>403 Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p>You don't have permission to access this resource.</p>\n</body></html>\n'. Retrying attempt 1 in 2 seconds."}
{"type":"log","@timestamp":"2022-01-07T00:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS. took: 13ms."}
{"type":"log","@timestamp":"2022-01-07T00:21:33+00:00","tags":["error","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] Action failed with '<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>403 Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p>You don't have permission to access this resource.</p>\n</body></html>\n'. Retrying attempt 2 in 4 seconds."}
{"type":"log","@timestamp":"2022-01-07T00:21:33+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS. took: 2027ms."}
{"type":"log","@timestamp":"2022-01-07T00:21:33+00:00","tags":["error","savedobjects-service"],"pid":1,"message":"[.kibana] Action failed with '<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>403 Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p>You don't have permission to access this resource.</p>\n</body></html>\n'. Retrying attempt 2 in 4 seconds."}

The only change I have made for the Elasticsearch.yml config file is that I added the "xpack.security.enabled: false". This is due to our implementation in which we have a front-end apache-proxy that handle basic authentication and therefore we choose to disable it in Elasticsearch.

Note that we use kubernetes to deploy multiple instances of Elasticsearch cluster pods and kibana pods. All Elasticsearch pods come up normal. The kibana pods that failed to come up will have the 403 error as shown above.

Also, output for GET _cat/indices/.kibana*:

green open .kibana_task_manager_7.16.2_001 zVI70QI4RV-kpGT0Wt7nWQ 1 1 17 2 77.7kb 38.8kb
green open .kibana_7.16.2_001              -oplNmORQom9owJbQiKiYQ 1 1 66 5  4.7mb  2.3mb
green open .kibana-event-log-7.16.2-000001 aUIzudR1QBKre1qKHt7Zuw 1 1  2 0 12.9kb  6.4kb

Is this issue related to xpack.security.enabled setting or mismatch of .kibana vs .kibana_7.16.2_001 indices or something else?

Kindly request for help.

2

That looks like there's a proxy issue.

3

I don't believe this is a proxy issue. With kubernetes multiple pods implementation of kibana, the first kibana pod can always come up, subsequent pods will have this issue.

The first kibana pod logs will show this:

{"type":"log","@timestamp":"2022-01-06T18:51:01+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2022-01-06T18:51:02+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] INIT -> CREATE_NEW_TARGET. took: 224ms."}
{"type":"log","@timestamp":"2022-01-06T18:51:02+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] INIT -> CREATE_NEW_TARGET. took: 227ms."}
{"type":"log","@timestamp":"2022-01-06T18:51:03+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] CREATE_NEW_TARGET -> MARK_VERSION_INDEX_READY. took: 1057ms."}
{"type":"log","@timestamp":"2022-01-06T18:51:03+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] CREATE_NEW_TARGET -> MARK_VERSION_INDEX_READY. took: 1055ms."}
{"type":"log","@timestamp":"2022-01-06T18:51:03+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] MARK_VERSION_INDEX_READY -> DONE. took: 9ms."}
{"type":"log","@timestamp":"2022-01-06T18:51:03+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] Migration completed after 1290ms"}
{"type":"log","@timestamp":"2022-01-06T18:51:03+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] MARK_VERSION_INDEX_READY -> DONE. took: 7ms."}
{"type":"log","@timestamp":"2022-01-06T18:51:03+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] Migration completed after 1289ms"}

Subsequent kibana failing pod(s) logs will show this:

{"type":"log","@timestamp":"2022-01-07T05:21:30+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 28ms."}
{"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 5ms."}
{"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 11ms."}
{"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 4ms."}
{"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["error","savedobjects-service"],"pid":1,"message":"[.kibana] Action failed with '<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>403 Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p>You don't have permission to access this resource.</p>\n</body></html>\n'. Retrying attempt 1 in 2 seconds."}
{"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS. took: 57ms."}
{"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 137ms."}
{"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 6ms."}
{"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 9ms."}
{"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 5ms."}
{"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["error","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] Action failed with '<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>403 Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p>You don't have permission to access this resource.</p>\n</body></html>\n'. Retrying attempt 1 in 2 seconds."}
{"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS. took: 14ms."}

It seems that subsequent failed pods that try to come up will go thru a different code path.

4

That is the only error in your log (another one a little earlier), and that response of the 403 is not something that Kibana or Elasticsearch will display with that included HTML. Which is why I think it's something else.

5

I wonder if this has something to do with the alias since .kibana and .kibana_task_manager is the alias for .kibana_7.16.2_001 and .kibana_task_manager_7.16.2_001 respectively. The alias does not have permission to access the document? Granting privileges for indices and aliases | Elasticsearch Guide [6.8] | Elastic

6

Some requests to Elasticsearch succeed, but the UPDATE_TARGET_MAPPINGS step fails. During this step Kibana will call PUT /.kibana_7.16.2_001/_mapping and for some reason this request is rejected.

It might be that Elasticsearch returns a 403 but that your proxy then renders an HTML page for this error.

7

Thanks Rudolf,

I checked the Elasticsearch logs but could not find the _mapping request being received. Our Elasticsearch proxy does deny request such as _update_by_query, but not the _mapping call.

8

Not sure why the _mapping request isn't showing up, but the very next step after the _mapping api call is to call _update_by_query so you would have to give Kibana permission to call that API

9

Do you know why Kibana call _udpate_by_query? I can try removing the deny for that api to see what happen.

10

There's a brief description here kibana/README.md at main · elastic/kibana · GitHub

For the full details about why the migration algorithm works the way it does see: kibana/0013_saved_object_migrations.md at main · elastic/kibana · GitHub

11

Thanks for the info. I really appreciate it. After letting Kibana to call the _update_by_query api, my issue is resolved.

1 Like
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.