Hi,
I have a question regarding on how to update preconfigured agent policies residing in kibana.yml.
We deploy on ECK and our stack is version 8.8.1.
The first time we deploy kibana the preconfigured policies residing under xpack.fleet.agentPolicies get initialised and work fantastically.
After the first time any changes made to kibana.yml are propagated by the operator to the kibana-kb-config secret (kibana.yml), but those changes are not reflected in the GUI and the agents are not being updated with the new policy.
Restarting kibana does nothing, Kibana says no policies need updating:
Line 18: [2023-07-06T13:11:40.746+00:00][DEBUG][plugins.fleet] Initializing plugin
Line 86: [2023-07-06T13:11:43.847+00:00][INFO ][plugins.fleet] Task Fleet-Usage-Sender-1.1.0 scheduled with interval 1h
Line 94: [2023-07-06T13:11:46.561+00:00][DEBUG][plugins.fleet.telemetry_events] Starting local task
Line 95: [2023-07-06T13:11:46.565+00:00][INFO ][plugins.fleet] Task Fleet-Usage-Logger-Task scheduled with interval 5m
Line 98: [2023-07-06T13:11:46.678+00:00][INFO ][plugins.fleet] Beginning fleet setup
Line 100: [2023-07-06T13:11:46.736+00:00][DEBUG][plugins.fleet] Setting Fleet server config
Line 114: [2023-07-06T13:11:46.950+00:00][DEBUG][plugins.fleet] Setting up Fleet download source
Line 115: [2023-07-06T13:11:46.959+00:00][DEBUG][plugins.fleet] Setting up Proxy
Line 117: [2023-07-06T13:11:46.965+00:00][DEBUG][plugins.fleet] Setting up Fleet Sever Hosts
Line 118: [2023-07-06T13:11:46.976+00:00][DEBUG][plugins.fleet] Setting up Fleet outputs
Line 124: [2023-07-06T13:11:47.064+00:00][DEBUG][plugins.fleet] Setting up Fleet Elasticsearch assets
Line 125: [2023-07-06T13:11:47.065+00:00][DEBUG][plugins.fleet] Creating Fleet component template and ingest pipeline
Line 127: [2023-07-06T13:11:47.080+00:00][DEBUG][plugins.fleet] Ensuring file upload write indices for elastic_agent
Line 128: [2023-07-06T13:11:47.144+00:00][DEBUG][plugins.fleet] Setting up initial Fleet packages
Line 130: [2023-07-06T13:11:48.034+00:00][DEBUG][plugins.fleet] kicking off bulk install of fleet_server, elastic_agent, iis, system, windows, apm, log, synthetics
Line 131: [2023-07-06T13:11:48.120+00:00][DEBUG][plugins.fleet] Running required package policies upgrades for managed policies
Line 132: [2023-07-06T13:11:48.156+00:00][DEBUG][plugins.fleet] Upgrade Fleet package install versions
Line 133: [2023-07-06T13:11:48.168+00:00][DEBUG][plugins.fleet] Generating key pair for message signing
Line 134: [2023-07-06T13:11:48.219+00:00][DEBUG][plugins.fleet] Upgrade Agent policy schema version
Line 135: [2023-07-06T13:11:48.235+00:00][DEBUG][plugins.fleet] Found 0 outdated agent policies
Line 136: [2023-07-06T13:11:48.235+00:00][DEBUG][plugins.fleet] Setting up Fleet enrollment keys
Line 137: [2023-07-06T13:11:48.278+00:00][INFO ][plugins.fleet] Fleet setup completed
Line 143: [2023-07-06T13:12:25.922+00:00][DEBUG][plugins.fleet] setting file list to the cache for log-2.0.0
Line 144: [2023-07-06T13:12:25.922+00:00][TRACE][plugins.fleet] ["log-2.0.0/LICENSE.txt","log-2.0.0/LICENSE.txt","log-2.0.0/agent/input/input.yml.hbs","log-2.0.0/agent/input/input.yml.hbs","log-2.0.0/changelog.yml","log-2.0.0/changelog.yml","log-2.0.0/docs/README.md","log-2.0.0/docs/README.md","log-2.0.0/fields/agent.yml","log-2.0.0/fields/agent.yml","log-2.0.0/fields/base-fields.yml","log-2.0.0/fields/base-fields.yml","log-2.0.0/img/icon.svg","log-2.0.0/img/icon.svg","log-2.0.0/manifest.yml","log-2.0.0/manifest.yml"]
Line 145: [2023-07-06T13:12:25.926+00:00][DEBUG][plugins.fleet] setting package info to the cache for log-2.0.0
Line 146: [2023-07-06T13:12:25.926+00:00][TRACE][plugins.fleet] {"name":"log","version":"2.0.0","description":"Collect custom logs with Elastic Agent.","title":"Custom Logs","format_version":"2.6.0","owner":{"github":"elastic/elastic-agent-data-plane"},"type":"input","categories":["custom","custom_logs"],"conditions":{"kibana.version":"^8.8.0"},"icons":[{"src":"/img/icon.svg","type":"image/svg+xml"}],"policy_templates":[{"name":"logs","title":"Custom log file","description":"Collect your custom log files.","multiple":true,"input":"logfile","type":"logs","template_path":"input.yml.hbs","vars":[{"name":"paths","required":true,"title":"Log file path","description":"Path to log files to be collected","type":"text","multi":true},{"name":"data_stream.dataset","required":true,"title":"Dataset name","description":"Set the name for your dataset. Changing the dataset will send the data to a different index. You can't use `-` in the name of a dataset and only valid characters for [Elasticsearch index names](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html).\n","type":"text"},{"name":"tags","type":"text","title":"Tags","description":"Tags to include in the published event","multi":true,"show_user":false},{"name":"processors","type":"yaml","title":"Processors","multi":false,"required":false,"show_user":false,"description":"Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details."},{"name":"custom","title":"Custom configurations","description":"Here YAML configuration options can be used to be added to your configuration. Be careful using this as it might break your configuration file.\n","type":"yaml","default":""}]}],"readme":"/package/log/2.0.0/docs/README.md","release":"ga"}
Line 147: [2023-07-06T13:12:25.926+00:00][DEBUG][plugins.fleet] retrieved installed package log-2.0.0 from ES
Line 148: [2023-07-06T13:12:26.523+00:00][DEBUG][plugins.fleet] retrieved installed package log-2.0.0 from cache
Line 149: [2023-07-06T13:12:40.801+00:00][DEBUG][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":5,"healthy":3,"unhealthy":1,"offline":1,"inactive":0,"unenrolled":7,"total_all_statuses":12,"updating":0},"fleet_server":{"total_enrolled":1,"healthy":1,"unhealthy":0,"offline":0,"updating":0,"total_all_statuses":1,"num_host_urls":2}}
I found in this issue which forces kibana to re-read specific policies but has the major drawback that it also deletes any existing Fleet API keys and unenrolls all agents.
Even if is_managed: true the policy does not get updated and we cannot enroll agents anymore, at least not from the GUI.
How should we approach this?
Thanks for your support.
George.