xpack.encryptedSavedObjects.encryptionKey value not working in Kibana

Elastic Stack Kibana
1

Hello,

I recently upgraded from 7.17 to 8.14 with both Elasticsearch and Kibana. One problem I'm experiencing is I am no longer able to access any of my rules and connectors which were previously notifying me of changes based on Elasticsearch queries.

Below is my Kibana.yml file:

elasticsearch.username: "kibana_system"
elasticsearch.password: "password"
xpack.reporting.encryptionKey: myReportingKey
xpack.security.encryptionKey: myKey
xpack.encryptedSavedObjects.encryptionKey: myKey

These are the log messages I am getting in my logfile:

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-06-08T16:44:10.149-04:00","message":"Failed to poll for work: ResponseError: {\"took\":3,\"timed_out\":false,\"total\":10,\"updated\":0
,\"deleted\":0,\"batches\":1,\"version_conflicts\":0,\"noops\":1,\"retries\":{\"bulk\":0,\"search\":0},\"throttled_millis\":0,\"requests_per_second\":-1,\"throttled_until_millis\":0,\"failures\":[{\"index\":\".kibana_task_manager_7.17.13
_001\",\"id\":\"task:b2b27ca0-c081-11ee-822f-55282727ff2c\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_task_manager_7.17.13_001] blocked by: [FORBIDDEN/8/index write (api)];\"},\"status\":403},{\"index\":
\".kibana_task_manager_7.17.13_001\",\"id\":\"task:1123d330-240d-11ef-8308-f5dcbf4257f2\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_task_manager_7.17.13_001] blocked by: [FORBIDDEN/8/index write (api)];\
"},\"status\":403},{\"index\":\".kibana_task_manager_7.17.13_001\",\"id\":\"task:endpoint:user-artifact-packager:1.0.0\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_task_manager_7.17.13_001] blocked by: [F
ORBIDDEN/8/index write (api)];\"},\"status\":403},{\"index\":\".kibana_task_manager_7.17.13_001\",\"id\":\"task:endpoint:complete-external-response-actions-1.0.0\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kiba
na_task_manager_7.17.13_001] blocked by: [FORBIDDEN/8/index write (api)];\"},\"status\":403},{\"index\":\".kibana_task_manager_7.17.13_001\",\"id\":\"task:c6930c30-0c99-11ef-8708-916653fddd74\",\"cause\":{\"type\":\"cluster_block_excepti
on\",\"reason\":\"index [.kibana_task_manager_7.17.13_001] blocked by: [FORBIDDEN/8/index write (api)];\"},\"status\":403},{\"index\":\".kibana_task_manager_7.17.13_001\",\"id\":\"task:e76a2200-0c89-11ef-8708-916653fddd74\",\"cause\":{\"
type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_task_manager_7.17.13_001] blocked by: [FORBIDDEN/8/index write (api)];\"},\"status\":403},{\"index\":\".kibana_task_manager_7.17.13_001\",\"id\":\"task:75cb4af0-0c7c-11ef-870
8-916653fddd74\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_task_manager_7.17.13_001] blocked by: [FORBIDDEN/8/index write (api)];\"},\"status\":403},{\"index\":\".kibana_task_manager_7.17.13_001\",\"id\"
:\"task:8f9d2450-1f80-11ef-aaaa-3b4e665a98c8\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_task_manager_7.17.13_001] blocked by: [FORBIDDEN/8/index write (api)];\"},\"status\":403},{\"index\":\".kibana_tas
k_manager_7.17.13_001\",\"id\":\"task:Fleet-Metrics-Task:1.1.1\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_task_manager_7.17.13_001] blocked by: [FORBIDDEN/8/index write (api)];\"},\"status\":403}]}","lo
g":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":54745,"uptime":801.450059669},"trace":{"id":"5cae71152747638e17e9f88d39bf18b5"},"transaction":{"id":"462ec399e47479c0"}}

{"tags":[".es-query","e71b17a0-0c89-11ef-8708-916653fddd74","rule-run-failed"],"error":{"stack_trace":"Error: Unable to create alerts client because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.\n    at Object.getRulesClientWithRequest (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/plugin.js:355:15)\n    at validateRuleAndCreateFakeRequest (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/rule_loader.js:51:31)\n    at /usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:351:79\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at TaskRunnerTimer.runWithTimer (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner_timer.js:49:20)\n    at TaskRunner.prepareToRun (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:301:12)\n    at TaskRunner.run (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:524:33)\n    at TaskManagerRunner.run (/usr/share/kibana/node_modules/@kbn/task-manager-plugin/server/task_running/task_runner.js:308:22)"},"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-06-08T13:39:03.920-04:00","message":"Executing Rule default:.es-query:e71b17a0-0c89-11ef-8708-916653fddd74 has resulted in Error: Unable to create alerts client because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command. - Error: Unable to create alerts client because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.\n    at Object.getRulesClientWithRequest (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/plugin.js:355:15)\n    at validateRuleAndCreateFakeRequest (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/rule_loader.js:51:31)\n    at /usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:351:79\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at TaskRunnerTimer.runWithTimer (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner_timer.js:49:20)\n    at TaskRunner.prepareToRun (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:301:12)\n    at TaskRunner.run (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:524:33)\n    at TaskManagerRunner.run (/usr/share/kibana/node_modules/@kbn/task-manager-plugin/server/task_running/task_runner.js:308:22)","log":{"level":"ERROR","logger":"plugins.alerting.es-query"},"process":{"pid":52247,"uptime":1910.468272187},"span":{"id":"970a6447c2b4c9ef"},"trace":{"id":"45564587558c3cdba7f36a81f7240cbf"}}

These logs don't make any sense since I'm explicitly specifying my encryption key in my kibana.yml file. Please let me know what other information is needed to assist me with my case.

2

Hi @fruitbat,

Welcome! Have you confirmed you have specified the correct keys? Were these generated using bin/kibana-encryption-keys or another method?

3

Hi @carly.richmond ,

Thanks for your response. These keys were generated manually and are 32 characters long. The alerting was previously working in 7.17.13. There is one alert I am seeing in my status page for TaskManager on Kibana:

{
  "level": "degraded",
  "summary": "Task Manager is unhealthy - Reason: setting HealthStatus.Warning because assumedAverageRecurringRequiredThroughputPerMinutePerKibana (46.90069444444445) < capacityPerMinutePerKibana (2000)",
  "reported": true
}

When I check kibana/api/task_manager/_health I get this:

{
  "id": "9a731384-b9c9-49a6-82af-9c4430486797",
  "timestamp": "2024-06-10T12:42:09.535Z",
  "status": "warn",
  "reason": "setting HealthStatus.Warning because assumedAverageRecurringRequiredThroughputPerMinutePerKibana (46.90069444444445) < capacityPerMinutePerKibana (2000)",
  "last_update": "2024-06-10T12:42:08.181Z",
  "stats": {
    "configuration": {
      "timestamp": "2024-06-09T19:59:42.980Z",
      "value": {
        "request_capacity": 1000,
        "monitored_aggregated_stats_refresh_rate": 60000,
        "monitored_stats_running_average_window": 50,
        "monitored_task_execution_thresholds": {
          "custom": {},
          "default": {
            "error_threshold": 90,
            "warn_threshold": 80
          }
        },
        "poll_interval": 3000,
        "max_workers": 100
      },
      "status": "OK"
    },
    "runtime": {
      "timestamp": "2024-06-09T19:59:42.980Z",
      "value": {
        "polling": {
          "claim_duration": {
            "p50": null,
            "p90": null,
            "p95": null,
            "p99": null
          },
          "duration": {
            "p50": null,
            "p90": null,
            "p95": null,
            "p99": null
          },
          "claim_conflicts": {
            "p50": null,
            "p90": null,
            "p95": null,
            "p99": null
          },
          "claim_mismatches": {
            "p50": null,
            "p90": null,
            "p95": null,
            "p99": null
          },
          "result_frequency_percent_as_number": {
            "Failed": 0,
            "NoAvailableWorkers": 0,
            "NoTasksClaimed": 0,
            "RanOutOfCapacity": 0,
            "RunningAtCapacity": 0,
            "PoolFilled": 0
          },
          "persistence": {
            "recurring": 0,
            "non_recurring": 0
          }
        },
        "drift": {
          "p50": null,
          "p90": null,
          "p95": null,
          "p99": null
        },
        "drift_by_type": {},
        "load": {
          "p50": null,
          "p90": null,
          "p95": null,
          "p99": null
        },
        "execution": {
          "duration": {},
          "duration_by_persistence": {
            "recurring": {
              "p50": null,
              "p90": null,
              "p95": null,
              "p99": null
            },
            "non_recurring": {
              "p50": null,
              "p90": null,
              "p95": null,
              "p99": null
            },
            "ephemeral": {
              "p50": null,
              "p90": null,
              "p95": null,
              "p99": null
            }
          },
          "persistence": {
            "recurring": 0,
            "non_recurring": 0,
            "ephemeral": 0
          },
          "result_frequency_percent_as_number": {}
        }
      },
      "status": "OK"
    },
    "workload": {
      "timestamp": "2024-06-10T12:41:43.752Z",
      "value": {
        "count": 43,
        "task_types": {
          "alerting:.es-query": {
            "count": 6,
            "status": {
              "idle": 6
            }
          },
          "security:endpoint-diagnostics": {
            "count": 2,
            "status": {
              "idle": 2
            }
          },
          "Fleet-Metrics-Task": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "Fleet-Usage-Logger": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "Fleet-Usage-Sender": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "ML:saved-objects-sync": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "SLO:ORPHAN_SUMMARIES-CLEANUP-TASK": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "actions_telemetry": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "alerting_health_check": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "alerting_telemetry": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "alerts_invalidate_api_keys": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "apm-telemetry-task": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "cases-telemetry-task": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "cleanup_failed_action_executions": {
            "count": 1,
            "status": {
              "unrecognized": 1
            }
          },
          "dashboard_telemetry": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "endpoint:complete-external-response-actions": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "endpoint:metadata-check-transforms-task": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "endpoint:user-artifact-packager": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "fleet:check-deleted-files-task": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "lens_telemetry": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "observabilityAIAssistant:indexQueuedDocumentsTaskType": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "osquery:telemetry-configs": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "osquery:telemetry-packs": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "osquery:telemetry-saved-queries": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "reports:monitor": {
            "count": 1,
            "status": {
              "unrecognized": 1
            }
          },
          "search_sessions_cleanup": {
            "count": 1,
            "status": {
              "unrecognized": 1
            }
          },
          "search_sessions_expire": {
            "count": 1,
            "status": {
              "unrecognized": 1
            }
          },
          "search_sessions_monitor": {
            "count": 1,
            "status": {
              "unrecognized": 1
            }
          },
          "security:endpoint-meta-telemetry": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "security:telemetry-configuration": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "security:telemetry-detection-rules": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "security:telemetry-diagnostic-timelines": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "security:telemetry-filterlist-artifact": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "security:telemetry-lists": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "security:telemetry-prebuilt-rule-alerts": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "security:telemetry-timelines": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "session_cleanup": {
            "count": 1,
            "status": {
              "idle": 1
            }
          }
        },
        "non_recurring": 43,
        "owner_ids": 0,
        "schedule": [
          [
            "3s",
            1
          ],
          [
            "10s",
            1
          ],
          [
            "15s",
            2
          ],
          [
            "30s",
            4
          ],
          [
            "60s",
            3
          ],
          [
            "1m",
            1
          ],
          [
            "5m",
            3
          ],
          [
            "15m",
            1
          ],
          [
            "45m",
            1
          ],
          [
            "1h",
            8
          ],
          [
            "3600s",
            2
          ],
          [
            "60m",
            2
          ],
          [
            "2h",
            1
          ],
          [
            "720m",
            2
          ],
          [
            "24h",
            6
          ],
          [
            "1d",
            3
          ]
        ],
        "overdue": 38,
        "overdue_non_recurring": 38,
        "estimated_schedule_density": [
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0
        ],
        "capacity_requirements": {
          "per_minute": 46,
          "per_hour": 53,
          "per_day": 25
        }
      },
      "status": "OK"
    },
    "capacity_estimation": {
      "status": "warn",
      "reason": "setting HealthStatus.Warning because assumedAverageRecurringRequiredThroughputPerMinutePerKibana (46.90069444444445) < capacityPerMinutePerKibana (2000)",
      "timestamp": "2024-06-10T12:42:09.535Z",
      "value": {
        "observed": {
          "observed_kibana_instances": 1,
          "max_throughput_per_minute_per_kibana": 2000,
          "max_throughput_per_minute": 2000,
          "minutes_to_drain_overdue": 38,
          "avg_recurring_required_throughput_per_minute": 47,
          "avg_recurring_required_throughput_per_minute_per_kibana": 47,
          "avg_required_throughput_per_minute": null,
          "avg_required_throughput_per_minute_per_kibana": null
        },
        "proposed": {
          "provisioned_kibana": null,
          "min_required_kibana": 1,
          "avg_recurring_required_throughput_per_minute_per_kibana": null,
          "avg_required_throughput_per_minute_per_kibana": null
        }
      }
    }
  }
}

It seems that all these tasks are failing, not just the alerts I have configured for. Let me know what your thoughts are!

4

Am I still running an older version of task manager? I restarted the process and while I no longer get the encryptedSavedObjects error message I do get a number of errors like this:

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-06-10T13:48:54.713-04:00","message":"Error scheduling Alerting-alerting_health_check, received index [.kibana_task_manager_7.17.13_001]
 blocked by: [FORBIDDEN/8/index write (api)];: cluster_block_exception\n\tRoot causes:\n\t\tcluster_block_exception: index [.kibana_task_manager_7.17.13_001] blocked by: [FORBIDDEN/8/index write (api)];","log":{"level":"ERROR","logger":"
plugins.alerting"},"process":{"pid":73422,"uptime":69.803523022},"trace":{"id":"70b04d49065d72be160e2fffa73b105c"},"transaction":{"id":"59da91ef14b2f2ec"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-06-10T13:48:54.716-04:00","message":"Error scheduling Alerts-alerts_invalidate_api_keys task, received index [.kibana_task_manager_7.17
.13_001] blocked by: [FORBIDDEN/8/index write (api)];: cluster_block_exception\n\tRoot causes:\n\t\tcluster_block_exception: index [.kibana_task_manager_7.17.13_001] blocked by: [FORBIDDEN/8/index write (api)];","log":{"level":"ERROR","l
ogger":"plugins.alerting.usage"},"process":{"pid":73422,"uptime":69.804092121},"trace":{"id":"70b04d49065d72be160e2fffa73b105c"},"transaction":{"id":"59da91ef14b2f2ec"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-06-10T13:48:54.727-04:00","message":"Error scheduling task, received error: ResponseError: index [.kibana_task_manager_7.17.13_001] blo
cked by: [FORBIDDEN/8/index write (api)];: cluster_block_exception\n\tRoot causes:\n\t\tcluster_block_exception: index [.kibana_task_manager_7.17.13_001] blocked by: [FORBIDDEN/8/index write (api)];","log":{"level":"ERROR","logger":"plug
ins.fleet"},"process":{"pid":73422,"uptime":69.804679749},"trace":{"id":"70b04d49065d72be160e2fffa73b105c"},"transaction":{"id":"59da91ef14b2f2ec"}}
{"name":"ResponseError","message":"Error scheduling task, received error: index [.kibana_task_manager_7.17.13_001] blocked by: [FORBIDDEN/8/index write (api)];: cluster_block_exception\n\tRoot causes:\n\t\tcluster_block_exception: index
[.kibana_task_manager_7.17.13_001] blocked by: [FORBIDDEN/8/index write (api)];","ecs":{"version":"8.11.0"},"@timestamp":"2024-06-10T13:48:54.730-04:00","log":{"level":"ERROR","logger":"plugins.fleet.fleet:check-deleted-files-task:1.0.1"
},"process":{"pid":73422,"uptime":69.805370867},"trace":{"id":"70b04d49065d72be160e2fffa73b105c"},"transaction":{"id":"59da91ef14b2f2ec"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-06-10T13:48:54.733-04:00","message":"Error scheduling task, received error: ResponseError: index [.kibana_task_manager_7.17.13_001] blo
cked by: [FORBIDDEN/8/index write (api)];: cluster_block_exception\n\tRoot causes:\n\t\tcluster_block_exception: index [.kibana_task_manager_7.17.13_001] blocked by: [FORBIDDEN/8/index write (api)];","log":{"level":"ERROR","logger":"plug
ins.fleet"},"process":{"pid":73422,"uptime":69.806325475},"trace":{"id":"70b04d49065d72be160e2fffa73b105c"},"transaction":{"id":"59da91ef14b2f2ec"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-06-10T13:48:54.737-04:00","message":"[task osquery:telemetry-packs:1.1.0]: error scheduling task, received index [.kibana_task_manager_
7.17.13_001] blocked by: [FORBIDDEN/8/index write (api)];: cluster_block_exception\n\tRoot causes:\n\t\tcluster_block_exception: index [.kibana_task_manager_7.17.13_001] blocked by: [FORBIDDEN/8/index write (api)];","log":{"level":"ERROR
","logger":"plugins.osquery.telemetry_events"},"process":{"pid":73422,"uptime":69.806908103},"trace":{"id":"70b04d49065d72be160e2fffa73b105c"},"transaction":{"id":"59da91ef14b2f2ec"}}
5

Ultimately, I just deleted all the old, restricted indexes that Kibana was attempting to write to and then had to allow my account access to create new restricted indexes. It appears the 403 errors were all access related.

1 Like