# Fail to start Kibana 7.16.2 **URL:** \ **Category:** Kibana\ **Created:** [January 7, 2022, 1:30am UTC](https://discuss.elastic.co/t/fail-to-start-kibana-7-16-2/293698 "2022-01-07T01:30:01Z")\ **Posts on this page:** 12\ **Page:** 1

**Author:** ![chilo](https://avatars.discourse-cdn.com/v4/letter/c/7cd45c/32.png) [@chilo](https://discuss.elastic.co/u/chilo)\ **Post date:** [January 7, 2022, 1:30am UTC](https://discuss.elastic.co/t/fail-to-start-kibana-7-16-2/293698/1 "2022-01-07T01:30:01Z")

I have Elasticsearch and kibana running on version 7.9.2 for a very long time without issues. Lately, when I started to deploy Elasticsearch and kibana 7.16.2, I see the 403 error below: ```auto {"type":"log","@timestamp":"2022-01-07T00:21:30+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"Starting saved objects migrations"} {"type":"log","@timestamp":"2022-01-07T00:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 34ms."} {"type":"log","@timestamp":"2022-01-07T00:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 4ms."} {"type":"log","@timestamp":"2022-01-07T00:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 11ms."} {"type":"log","@timestamp":"2022-01-07T00:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 4ms."} {"type":"log","@timestamp":"2022-01-07T00:21:31+00:00","tags":["error","savedobjects-service"],"pid":1,"message":"[.kibana] Action failed with '\n\n403 Forbidden\n\n

Forbidden

You don't have permission to access this resource.

\n\n'. Retrying attempt 1 in 2 seconds."} {"type":"log","@timestamp":"2022-01-07T00:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS. took: 85ms."} {"type":"log","@timestamp":"2022-01-07T00:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 149ms."} {"type":"log","@timestamp":"2022-01-07T00:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 7ms."} {"type":"log","@timestamp":"2022-01-07T00:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 25ms."} {"type":"log","@timestamp":"2022-01-07T00:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 5ms."} {"type":"log","@timestamp":"2022-01-07T00:21:31+00:00","tags":["error","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] Action failed with '\n\n403 Forbidden\n\n

Forbidden

You don't have permission to access this resource.

Forbidden

You don't have permission to access this resource.

\n\n'. Retrying attempt 2 in 4 seconds."} {"type":"log","@timestamp":"2022-01-07T00:21:33+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS. took: 2027ms."} {"type":"log","@timestamp":"2022-01-07T00:21:33+00:00","tags":["error","savedobjects-service"],"pid":1,"message":"[.kibana] Action failed with '\n\n403 Forbidden\n\n

Forbidden

You don't have permission to access this resource.

\n\n'. Retrying attempt 2 in 4 seconds."} ``` The only change I have made for the Elasticsearch.yml config file is that I added the "xpack.security.enabled: false". This is due to our implementation in which we have a front-end apache-proxy that handle basic authentication and therefore we choose to disable it in Elasticsearch. Note that we use kubernetes to deploy multiple instances of Elasticsearch cluster pods and kibana pods. All Elasticsearch pods come up normal. The kibana pods that failed to come up will have the 403 error as shown above. Also, output for GET \_cat/indices/.kibana\*: ```auto green open .kibana_task_manager_7.16.2_001 zVI70QI4RV-kpGT0Wt7nWQ 1 1 17 2 77.7kb 38.8kb green open .kibana_7.16.2_001 -oplNmORQom9owJbQiKiYQ 1 1 66 5 4.7mb 2.3mb green open .kibana-event-log-7.16.2-000001 aUIzudR1QBKre1qKHt7Zuw 1 1 2 0 12.9kb 6.4kb ``` Is this issue related to xpack.security.enabled setting or mismatch of .kibana vs .kibana\_7.16.2\_001 indices or something else? Kindly request for help. ---

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 9, 2022, 9:56pm UTC](https://discuss.elastic.co/t/fail-to-start-kibana-7-16-2/293698/2 "2022-01-09T21:56:09Z")

That looks like there's a proxy issue. ---

**Author:** ![chilo](https://avatars.discourse-cdn.com/v4/letter/c/7cd45c/32.png) [@chilo](https://discuss.elastic.co/u/chilo)\ **Post date:** [January 11, 2022, 10:00pm UTC](https://discuss.elastic.co/t/fail-to-start-kibana-7-16-2/293698/3 "2022-01-11T22:00:22Z")

I don't believe this is a proxy issue. With kubernetes multiple pods implementation of kibana, the first kibana pod can always come up, subsequent pods will have this issue. The first kibana pod logs will show this: ```auto {"type":"log","@timestamp":"2022-01-06T18:51:01+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"Starting saved objects migrations"} {"type":"log","@timestamp":"2022-01-06T18:51:02+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] INIT -> CREATE_NEW_TARGET. took: 224ms."} {"type":"log","@timestamp":"2022-01-06T18:51:02+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] INIT -> CREATE_NEW_TARGET. took: 227ms."} {"type":"log","@timestamp":"2022-01-06T18:51:03+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] CREATE_NEW_TARGET -> MARK_VERSION_INDEX_READY. took: 1057ms."} {"type":"log","@timestamp":"2022-01-06T18:51:03+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] CREATE_NEW_TARGET -> MARK_VERSION_INDEX_READY. took: 1055ms."} {"type":"log","@timestamp":"2022-01-06T18:51:03+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] MARK_VERSION_INDEX_READY -> DONE. took: 9ms."} {"type":"log","@timestamp":"2022-01-06T18:51:03+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] Migration completed after 1290ms"} {"type":"log","@timestamp":"2022-01-06T18:51:03+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] MARK_VERSION_INDEX_READY -> DONE. took: 7ms."} {"type":"log","@timestamp":"2022-01-06T18:51:03+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] Migration completed after 1289ms"} ``` Subsequent kibana failing pod(s) logs will show this: ```auto {"type":"log","@timestamp":"2022-01-07T05:21:30+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"Starting saved objects migrations"} {"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 28ms."} {"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 5ms."} {"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 11ms."} {"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 4ms."} {"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["error","savedobjects-service"],"pid":1,"message":"[.kibana] Action failed with '\n\n403 Forbidden\n\n

Forbidden

You don't have permission to access this resource.

\n\n'. Retrying attempt 1 in 2 seconds."} {"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS. took: 57ms."} {"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 137ms."} {"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 6ms."} {"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 9ms."} {"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 5ms."} {"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["error","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] Action failed with '\n\n403 Forbidden\n\n

Forbidden

You don't have permission to access this resource.

\n\n'. Retrying attempt 1 in 2 seconds."} {"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["info","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS. took: 14ms."} ``` It seems that subsequent failed pods that try to come up will go thru a different code path. ---

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 11, 2022, 10:05pm UTC](https://discuss.elastic.co/t/fail-to-start-kibana-7-16-2/293698/4 "2022-01-11T22:05:13Z")

> [@chilo](#): > > ```auto > {"type":"log","@timestamp":"2022-01-07T05:21:31+00:00","tags":["error","savedobjects-service"],"pid":1,"message":"[.kibana_task_manager] Action failed with '\n\n403 Forbidden\n\n

Forbidden

You don't have permission to access this resource.

\n\n'. Retrying attempt 1 in 2 seconds."} > > ``` That is the only error in your log (another one a little earlier), and that response of the 403 is not something that Kibana or Elasticsearch will display with that included HTML. Which is why I think it's something else. ---

**Author:** ![chilo](https://avatars.discourse-cdn.com/v4/letter/c/7cd45c/32.png) [@chilo](https://discuss.elastic.co/u/chilo)\ **Post date:** [January 11, 2022, 10:32pm UTC](https://discuss.elastic.co/t/fail-to-start-kibana-7-16-2/293698/5 "2022-01-11T22:32:37Z")

I wonder if this has something to do with the alias since .kibana and .kibana\_task\_manager is the alias for .kibana\_7.16.2\_001 and .kibana\_task\_manager\_7.16.2\_001 respectively. The alias does not have permission to access the document? [Granting privileges for indices and aliases | Elasticsearch Guide [6.8] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/securing-aliases.html) ---

**Author:** ![rudolf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rudolf/32/51200_2.png) [@rudolf](https://discuss.elastic.co/u/rudolf)\ **Post date:** [January 13, 2022, 10:12pm UTC](https://discuss.elastic.co/t/fail-to-start-kibana-7-16-2/293698/6 "2022-01-13T22:12:27Z")

Some requests to Elasticsearch succeed, but the `UPDATE_TARGET_MAPPINGS` step fails. During this step Kibana will call `PUT /.kibana_7.16.2_001/_mapping` and for some reason this request is rejected. It might be that Elasticsearch returns a 403 but that your proxy then renders an HTML page for this error. ---

**Author:** ![chilo](https://avatars.discourse-cdn.com/v4/letter/c/7cd45c/32.png) [@chilo](https://discuss.elastic.co/u/chilo)\ **Post date:** [January 17, 2022, 10:13pm UTC](https://discuss.elastic.co/t/fail-to-start-kibana-7-16-2/293698/7 "2022-01-17T22:13:18Z")

Thanks Rudolf, I checked the Elasticsearch logs but could not find the \_mapping request being received. Our Elasticsearch proxy does deny request such as \_update\_by\_query, but not the \_mapping call. ---

**Author:** ![rudolf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rudolf/32/51200_2.png) [@rudolf](https://discuss.elastic.co/u/rudolf)\ **Post date:** [January 18, 2022, 2:58pm UTC](https://discuss.elastic.co/t/fail-to-start-kibana-7-16-2/293698/8 "2022-01-18T14:58:40Z")

Not sure why the \_mapping request isn't showing up, but the very next step after the \_mapping api call is to call \_update\_by\_query so you would have to give Kibana permission to call that API ---

**Author:** ![chilo](https://avatars.discourse-cdn.com/v4/letter/c/7cd45c/32.png) [@chilo](https://discuss.elastic.co/u/chilo)\ **Post date:** [January 18, 2022, 3:26pm UTC](https://discuss.elastic.co/t/fail-to-start-kibana-7-16-2/293698/9 "2022-01-18T15:26:56Z")

Do you know why Kibana call \_udpate\_by\_query? I can try removing the deny for that api to see what happen. ---

**Author:** ![rudolf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rudolf/32/51200_2.png) [@rudolf](https://discuss.elastic.co/u/rudolf)\ **Post date:** [January 24, 2022, 1:51pm UTC](https://discuss.elastic.co/t/fail-to-start-kibana-7-16-2/293698/10 "2022-01-24T13:51:49Z")

There's a brief description here [kibana/README.md at main · elastic/kibana · GitHub](https://github.com/elastic/kibana/blob/main/src/core/server/saved_objects/migrations/README.md#L378-L383) For the full details about why the migration algorithm works the way it does see: [kibana/0013\_saved\_object\_migrations.md at main · elastic/kibana · GitHub](https://github.com/elastic/kibana/blob/main/legacy_rfcs/text/0013_saved_object_migrations.md) ---

**Author:** ![chilo](https://avatars.discourse-cdn.com/v4/letter/c/7cd45c/32.png) [@chilo](https://discuss.elastic.co/u/chilo)\ **Post date:** [January 26, 2022, 12:08am UTC](https://discuss.elastic.co/t/fail-to-start-kibana-7-16-2/293698/11 "2022-01-26T00:08:28Z")

Thanks for the info. I really appreciate it. After letting Kibana to call the \_update\_by\_query api, my issue is resolved. ---

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 23, 2022, 12:08am UTC](https://discuss.elastic.co/t/fail-to-start-kibana-7-16-2/293698/12 "2022-02-23T00:08:37Z")

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.