./bin/kibana-keystore add elastic.apm.secretToken does nothing

[learningelastic]

I can't seem to get the ./bin/kibana-keystore to work for the field elastic.apm.secretToken.

I started by making sure Kibana can work with APM. This kibana.yml file works perfectly:

elastic:
  apm:
    active: true
    serverUrl: "http://fleet-server:8200"
    secretToken: "supersecrettoken"
server.host: "0.0.0.0"
telemetry.enabled: "true"
xpack.fleet.packages:
  - name: fleet_server
    version: latest
  - name: system
    version: latest
  - name: elastic_agent
    version: latest
  - name: apm
    version: latest
xpack.fleet.agentPolicies:
  - name: Fleet-Server-Policy
    id: fleet-server-policy
    namespace: default
    monitoring_enabled:
      - logs
      - metrics
    package_policies:
      - name: fleet_server-1
        package:
          name: fleet_server
      - name: system-1
        package:
          name: system
      - name: elastic_agent-1
        package:
          name: elastic_agent
      - name: apm-1
        package:
          name: apm
        inputs:
        - type: apm
          enabled: true
          vars:
          - name: host
            value: 0.0.0.0:8200
          - name: secret_token
            value: "supersecrettoken"

After I configure Fleet Server, I can see kibana listed as a service in my web browser at https://192.168.0.22:5601/app/apm/services.

However, if I delete just the one line elastic.apm.secretToken, and repeat the whole set up with

echo "supersecrettoken"| /usr/share/kibana/bin/kibana-keystore add elastic.apm.secretToken -x;

Then this breaks the connection between Kibana and APM. I'm pretty confident that my ./bin/kibana-keystore is working for every other field. It is properly setting fields like xpack.security.encryptionKey, xpack.encryptedSavedObjects.encryptionKey, xpack.reporting.encryptionKey.

The only fields it can't seem to set are:

 /usr/share/kibana/bin/kibana-keystore add elastic.apm.secretToken
/usr/share/kibana/bin/kibana-keystore add xpack.fleet.agentPolicies[0].package_policies[3].inputs[0].vars[1].value

Is this a known issue?

Kibana version:

8.8.2

Elasticsearch version:

8.8.2

APM Server version:

8.8.2

Original install method (e.g. download page, yum, deb, from source, etc.) and version:

I'm using docker-compose and I'm starting from this github project:

GitHub - elkninja/elastic-stack-docker-part-two

Getting started with the Elastic Stack and Docker Compose: Part 2

Fresh install or upgraded from other version?

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

No, i'm just trying to use kibana keystore to save APM secret Tokens.