Apm-server without ElasticSearch - refresh cache error: context deadline exceeded

APM Server version: 8.17.0-1

Problem:
When starting the apm-server as standalone binary, no fleet, no Elasticsearch output, but output configured to Kafka, it keeps reporting errors about agentcfg: "failed to clear scroll: context deadline exceeded" and "refresh cache error: context deadline exceeded". From what i understand it's still trying to connect to Elasticsearch, hence I have tried disabling it via the apm-server.agent.config.elasticsearch.enabled: false setting. All without luck.

Any ideas?

Startup: via systemd service

/usr/share/apm-server/bin/apm-server --environment systemd --path.config /var/appdata/apm-server/config

Config: apm-server.yml

apm-server:
  host: 0.0.0.0:8200
  agent.config.elasticsearch.enabled: false

output.kafka:
  enabled: true
  hosts: kafa-host:9094
  topic: apm
  compression: lz4
  required_acks: 1
  ssl.certificate_authorities: my.truststore

path.home: /var/appdata/apm-server
path.config: ${path.home}/config
path.data: ${path.home}/data
path.logs: ${path.home}/logs

logging.level: debug

Logs:

Apr 23 16:54:20 my-server apm-server[3105114]: {"log.level":"warn","@timestamp":"2025-04-23T16:54:20.730+0200","log.logger":"agentcfg","log.origin":{"function":"github.com/elastic/apm-server/internal/agentcfg.(*ElasticsearchFetcher).clearScroll","file.name":"agentcfg/elasticsearch.go","file.line":260},"message":"failed to clear scroll: context deadline exceeded","service.name":"apm-server","ecs.version":"1.6.0"}
Apr 23 16:54:20 my-server apm-server[3105114]: {"log.level":"error","@timestamp":"2025-04-23T16:54:20.731+0200","log.logger":"agentcfg","log.origin":{"function":"github.com/elastic/apm-server/internal/agentcfg.(*ElasticsearchFetcher).Run.func1","file.name":"agentcfg/elasticsearch.go","file.line":150},"message":"refresh cache error: context deadline exceeded","service.name":"apm-server","ecs.version":"1.6.0"}