"Forbidden" error when saving Dashboards

Elastic Stack Kibana
1

Hi there, I'm having a weird issue where Kibana reports "Error: Forbidden" when trying to save a Dashboard.
image

My logs in Kibana are looking like this:

{"type":"log","@timestamp":"2024-08-07T13:23:47+00:00","tags":["error","plugins","alerting"],"pid":1160624,"message":"error updating alert execution status for .es-query:44
117da0-7934-11ec-b7a9-715755f2a61d index [.kibana_7.16.1_001] blocked by: [FORBIDDEN/8/index write (api)];: cluster_block_exception: [cluster_block_exception] Reason: index
 [.kibana_7.16.1_001] blocked by: [FORBIDDEN/8/index write (api)];"}
{"type":"log","@timestamp":"2024-08-07T13:23:47+00:00","tags":["error","plugins","alerting"],"pid":1160624,"message":"error updating alert execution status for .es-query:78
5cad90-7935-11ec-b7a9-715755f2a61d index [.kibana_7.16.1_001] blocked by: [FORBIDDEN/8/index write (api)];: cluster_block_exception: [cluster_block_exception] Reason: index
 [.kibana_7.16.1_001] blocked by: [FORBIDDEN/8/index write (api)];"}
{"type":"log","@timestamp":"2024-08-07T13:23:47+00:00","tags":["error","plugins","alerting"],"pid":1160624,"message":"error updating alert execution status for .es-query:a0
671e40-9e44-11ec-b7a9-715755f2a61d index [.kibana_7.16.1_001] blocked by: [FORBIDDEN/8/index write (api)];: cluster_block_exception: [cluster_block_exception] Reason: index
 [.kibana_7.16.1_001] blocked by: [FORBIDDEN/8/index write (api)];"}
{"type":"log","@timestamp":"2024-08-07T13:23:47+00:00","tags":["error","plugins","alerting"],"pid":1160624,"message":"Executing Alert default:.es-query:a0671e40-9e44-11ec-b
7a9-715755f2a61d has resulted in Error: index [.kibana_7.16.1_001] blocked by: [FORBIDDEN/8/index write (api)];: cluster_block_exception: [cluster_block_exception] Reason:
index [.kibana_7.16.1_001] blocked by: [FORBIDDEN/8/index write (api)];, caused by: \"\""}
{"type":"log","@timestamp":"2024-08-07T13:23:50+00:00","tags":["error","plugins","dataEnhanced","data_enhanced"],"pid":1160624,"message":"search_sessions_cleanup Error whil
e deleting session 2943cb14-161a-4320-aa27-c34878973fe8: index [.kibana_7.16.1_001] blocked by: [FORBIDDEN/8/index write (api)];: cluster_block_exception: [cluster_block_ex
ception] Reason: index [.kibana_7.16.1_001] blocked by: [FORBIDDEN/8/index write (api)];"}
{"type":"log","@timestamp":"2024-08-07T13:23:50+00:00","tags":["error","plugins","dataEnhanced","data_enhanced"],"pid":1160624,"message":"search_sessions_cleanup Error whil
e deleting session 000ba87d-45da-41d5-902d-21d8f4a41eb2: index [.kibana_7.16.1_001] blocked by: [FORBIDDEN/8/index write (api)];: cluster_block_exception: [cluster_block_ex
ception] Reason: index [.kibana_7.16.1_001] blocked by: [FORBIDDEN/8/index write (api)];"}
{"type":"log","@timestamp":"2024-08-07T13:23:50+00:00","tags":["error","plugins","dataEnhanced","data_enhanced"],"pid":1160624,"message":"search_sessions_cleanup Error whil
e deleting session 63fc6905-1057-4b32-9b47-43707af6a46e: index [.kibana_7.16.1_001] blocked by: [FORBIDDEN/8/index write (api)];: cluster_block_exception: [cluster_block_ex
ception] Reason: index [.kibana_7.16.1_001] blocked by: [FORBIDDEN/8/index write (api)];"}
{"type":"log","@timestamp":"2024-08-07T13:23:50+00:00","tags":["error","plugins","dataEnhanced","data_enhanced"],"pid":1160624,"message":"search_sessions_cleanup Error whil
e deleting async_search FnIzMEpVRFdDUnFXaFFMbmZmeE9lUVEfbDNlNnp5NGJRTXVRSEFvcTVteFFMUTo2NTY1ODYyOA==: resource_not_found_exception: [resource_not_found_exception] Reason: F
nIzMEpVRFdDUnFXaFFMbmZmeE9lUVEfbDNlNnp5NGJRTXVRSEFvcTVteFFMUTo2NTY1ODYyOA=="}
{"type":"log","@timestamp":"2024-08-07T13:23:50+00:00","tags":["error","plugins","dataEnhanced","data_enhanced"],"pid":1160624,"message":"search_sessions_cleanup Error whil
e deleting async_search FjNLQXdNTGFkVGFTelVJOG1uUXE2OWceS3Znb0dGZWtTajJnLUVUeWRMck5vZzoxNTc3NzIx: resource_not_found_exception: [resource_not_found_exception] Reason: FjNLQ
XdNTGFkVGFTelVJOG1uUXE2OWceS3Znb0dGZWtTajJnLUVUeWRMck5vZzoxNTc3NzIx"}
{"type":"log","@timestamp":"2024-08-07T13:23:50+00:00","tags":["error","plugins","dataEnhanced","data_enhanced"],"pid":1160624,"message":"search_sessions_cleanup Error whil
e deleting async_search FnNqV0dzSXpuUXVhNEM4SldySEtNR2cfbDNlNnp5NGJRTXVRSEFvcTVteFFMUTo2NTUyOTA1OA==: resource_not_found_exception: [resource_not_found_exception] Reason: F
nNqV0dzSXpuUXVhNEM4SldySEtNR2cfbDNlNnp5NGJRTXVRSEFvcTVteFFMUTo2NTUyOTA1OA=="}

To give some context, on monday we started a new extended cluster, so the setup looks like 3 nodes in each DC, all of them in a single cluster (kind of a high availability scenario) and 2 kibanas, one per DC, looking to it's corresponding nodes. After these past days the reallocation of shards is almost finished, both kibanas are working great and see the same data. However, today when we were going to make some changes in a dashboard, we were reported the afordmention error.

I'm thinking the error might be due to some reallocation of shards in process, but I don't know. Is there anything I can do?

2

Hi @Jose_E,

What's the state of your disk space in your cluster? Looking through similar issues I wonder if it's a write block rather than an entitlements issue.

Can you check if it's a write block through the GET API:

GET https://localhost:9200/.kibana_7.16.1_001/_settings
3

Thanks for the answer, I thought the same that it might be a problem with space. After checking it was not and some restarts, the logs printed a very helpful line that said "The .kibana_task_manager alias is pointing to a newer version of Kibana: v7.17.21".

Turns out that after setting up the six nodes with the new cluster configuration, I forgot to update the Kibana at this DC, which still had an older version: 7.16.3. The other Kibana was up to date, though, and that's why it wasn't complaining.

It's all sorted out now, thank you.