How to update preconfigured fleet agent policies

Hi,

I have a question regarding on how to update preconfigured agent policies residing in kibana.yml.

We deploy on ECK and our stack is version 8.8.1.

The first time we deploy kibana the preconfigured policies residing under xpack.fleet.agentPolicies get initialised and work fantastically.

After the first time any changes made to kibana.yml are propagated by the operator to the kibana-kb-config secret (kibana.yml), but those changes are not reflected in the GUI and the agents are not being updated with the new policy.

Restarting kibana does nothing, Kibana says no policies need updating:

Line  18: [2023-07-06T13:11:40.746+00:00][DEBUG][plugins.fleet] Initializing plugin
	Line  86: [2023-07-06T13:11:43.847+00:00][INFO ][plugins.fleet] Task Fleet-Usage-Sender-1.1.0 scheduled with interval 1h
	Line  94: [2023-07-06T13:11:46.561+00:00][DEBUG][plugins.fleet.telemetry_events] Starting local task
	Line  95: [2023-07-06T13:11:46.565+00:00][INFO ][plugins.fleet] Task Fleet-Usage-Logger-Task scheduled with interval 5m
	Line  98: [2023-07-06T13:11:46.678+00:00][INFO ][plugins.fleet] Beginning fleet setup
	Line 100: [2023-07-06T13:11:46.736+00:00][DEBUG][plugins.fleet] Setting Fleet server config
	Line 114: [2023-07-06T13:11:46.950+00:00][DEBUG][plugins.fleet] Setting up Fleet download source
	Line 115: [2023-07-06T13:11:46.959+00:00][DEBUG][plugins.fleet] Setting up Proxy
	Line 117: [2023-07-06T13:11:46.965+00:00][DEBUG][plugins.fleet] Setting up Fleet Sever Hosts
	Line 118: [2023-07-06T13:11:46.976+00:00][DEBUG][plugins.fleet] Setting up Fleet outputs
	Line 124: [2023-07-06T13:11:47.064+00:00][DEBUG][plugins.fleet] Setting up Fleet Elasticsearch assets
	Line 125: [2023-07-06T13:11:47.065+00:00][DEBUG][plugins.fleet] Creating Fleet component template and ingest pipeline
	Line 127: [2023-07-06T13:11:47.080+00:00][DEBUG][plugins.fleet] Ensuring file upload write indices for elastic_agent
	Line 128: [2023-07-06T13:11:47.144+00:00][DEBUG][plugins.fleet] Setting up initial Fleet packages
	Line 130: [2023-07-06T13:11:48.034+00:00][DEBUG][plugins.fleet] kicking off bulk install of fleet_server, elastic_agent, iis, system, windows, apm, log, synthetics
	Line 131: [2023-07-06T13:11:48.120+00:00][DEBUG][plugins.fleet] Running required package policies upgrades for managed policies
	Line 132: [2023-07-06T13:11:48.156+00:00][DEBUG][plugins.fleet] Upgrade Fleet package install versions
	Line 133: [2023-07-06T13:11:48.168+00:00][DEBUG][plugins.fleet] Generating key pair for message signing
	Line 134: [2023-07-06T13:11:48.219+00:00][DEBUG][plugins.fleet] Upgrade Agent policy schema version
	Line 135: [2023-07-06T13:11:48.235+00:00][DEBUG][plugins.fleet] Found 0 outdated agent policies
	Line 136: [2023-07-06T13:11:48.235+00:00][DEBUG][plugins.fleet] Setting up Fleet enrollment keys
	Line 137: [2023-07-06T13:11:48.278+00:00][INFO ][plugins.fleet] Fleet setup completed
	Line 143: [2023-07-06T13:12:25.922+00:00][DEBUG][plugins.fleet] setting file list to the cache for log-2.0.0
	Line 144: [2023-07-06T13:12:25.922+00:00][TRACE][plugins.fleet] ["log-2.0.0/LICENSE.txt","log-2.0.0/LICENSE.txt","log-2.0.0/agent/input/input.yml.hbs","log-2.0.0/agent/input/input.yml.hbs","log-2.0.0/changelog.yml","log-2.0.0/changelog.yml","log-2.0.0/docs/README.md","log-2.0.0/docs/README.md","log-2.0.0/fields/agent.yml","log-2.0.0/fields/agent.yml","log-2.0.0/fields/base-fields.yml","log-2.0.0/fields/base-fields.yml","log-2.0.0/img/icon.svg","log-2.0.0/img/icon.svg","log-2.0.0/manifest.yml","log-2.0.0/manifest.yml"]
	Line 145: [2023-07-06T13:12:25.926+00:00][DEBUG][plugins.fleet] setting package info to the cache for log-2.0.0
	Line 146: [2023-07-06T13:12:25.926+00:00][TRACE][plugins.fleet] {"name":"log","version":"2.0.0","description":"Collect custom logs with Elastic Agent.","title":"Custom Logs","format_version":"2.6.0","owner":{"github":"elastic/elastic-agent-data-plane"},"type":"input","categories":["custom","custom_logs"],"conditions":{"kibana.version":"^8.8.0"},"icons":[{"src":"/img/icon.svg","type":"image/svg+xml"}],"policy_templates":[{"name":"logs","title":"Custom log file","description":"Collect your custom log files.","multiple":true,"input":"logfile","type":"logs","template_path":"input.yml.hbs","vars":[{"name":"paths","required":true,"title":"Log file path","description":"Path to log files to be collected","type":"text","multi":true},{"name":"data_stream.dataset","required":true,"title":"Dataset name","description":"Set the name for your dataset. Changing the dataset will send the data to a different index. You can't use `-` in the name of a dataset and only valid characters for [Elasticsearch index names](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html).\n","type":"text"},{"name":"tags","type":"text","title":"Tags","description":"Tags to include in the published event","multi":true,"show_user":false},{"name":"processors","type":"yaml","title":"Processors","multi":false,"required":false,"show_user":false,"description":"Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details."},{"name":"custom","title":"Custom configurations","description":"Here YAML configuration options can be used to be added to your configuration. Be careful using this as it might break your configuration file.\n","type":"yaml","default":""}]}],"readme":"/package/log/2.0.0/docs/README.md","release":"ga"}
	Line 147: [2023-07-06T13:12:25.926+00:00][DEBUG][plugins.fleet] retrieved installed package log-2.0.0 from ES
	Line 148: [2023-07-06T13:12:26.523+00:00][DEBUG][plugins.fleet] retrieved installed package log-2.0.0 from cache
	Line 149: [2023-07-06T13:12:40.801+00:00][DEBUG][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":5,"healthy":3,"unhealthy":1,"offline":1,"inactive":0,"unenrolled":7,"total_all_statuses":12,"updating":0},"fleet_server":{"total_enrolled":1,"healthy":1,"unhealthy":0,"offline":0,"updating":0,"total_all_statuses":1,"num_host_urls":2}}

I found in this issue which forces kibana to re-read specific policies but has the major drawback that it also deletes any existing Fleet API keys and unenrolls all agents.

Even if is_managed: true the policy does not get updated and we cannot enroll agents anymore, at least not from the GUI.

How should we approach this?

Thanks for your support.

George.

Hi @GeorgeGkinis

We currently do not support updating preconfigured agent policies this a known issue and hopefully we can propose a solution for that soon [Fleet] Preconfiguration API doesn't support updating / adding / removing package_policies · Issue #111401 · elastic/kibana · GitHub

In the mean time, you may use the our internal reset API that will apply your change

curl -XPOST https://<kibana url>/internal/fleet/reset_preconfigured_agent_policies/<your_preconfigured_policy_id> -u elastic:<password> -H 'kbn-xsrf: xyz' -H 'content-type: application/json'

Edit: just saw that you already got that API, I do not think we have a better solution currently

Thanks for your prompt answer!

Too bad we cannot add parameters to the API like :

  • unenroll_agents: false
  • cleanup_tokens: false
  • e.t.c.

It would cover a multitude of use cases.

Thanks again!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.