Heartbeat config files fail when path variables passed

[DPattee]

In order to get my Kibana https heartbeat working, I had to add the standard ssl blob. I used the same blob that I have in all my .yml's that lets everything successfully (mostly) talk over https:

  ssl.enabled: true
  ssl:
    certificate_authorities: ["${path.config}/elasticsearch-ca.pem"]
    verification_mode: "none"

But, that ended up causing the heartbeat to stop parsing that set of configs, instead spewing logs of:

2021-11-22T16:17:05.278-0800	ERROR	[reload]	cfgfile/list.go:69	Unable to hash given config: missing field accessing '0.ssl' (source:'/blahblah/elastic-configs/heartbeat/heartbeat-monitors/kibana.http.yml')

That error message was not informative beyond "something is wrong with the ssl section"... Some googling found a few other people with similar "unable to hash" errors, so I tried a few of the solutions.

The one that worked was replacing ${path.config}/Elasticsearch-ca.pem with the full hardcoded path to that file.

That works, as a temporary measure. But since that path isn't the same across all machines (and I use common config files that get synced to maintain my sanity) it isn't a good long term solution.

The common beat variables need to be properly parsed when they are in the module-specific ymls

[kvch (Noémi Ványi)]

Could you please share all of your configuration files? I have tried to reproduce your issue with a minimal configuration, but for me it is working as expected.

[DPattee]

Directory layout:

/usr/local/opt/heartbeat-full/<the app install>
/foo/elastic-configs/heartbeat/
/foo/elastic-configs/heartbeat/elasticsearch-ca.pem
/foo/elastic-configs/heartbeat/heartbeat-monitors/
/foo/elastic-configs/heartbeat/heartbeat-monitors/servers.icmp.yml
/foo/elastic-configs/heartbeat/heartbeat-monitors/kibana.http.yml

launch command:

/usr/local/opt/heartbeat-full/bin/heartbeat --path.config /foo/elastic-configs/heartbeat -c my-heartbeat-config.yml

Main config:

heartbeat.config.monitors:
  # Directory + glob pattern to search for configuration files
  path: ${path.config}/heartbeat-monitors/*.yml
  reload.enabled: true
  reload.period: 300s

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["headnode.hostname:9200"]
  protocol: "https"
  username: "heartbeat_user"
  password: "password"
  ssl:
    certificate_authorities: ["${path.config}/elasticsearch-ca.pem"]
    verification_mode: "none"

  max_retries: 5
  backoff.max: 120s
  timeout: 120

processors:
  - add_locale:
      format: offset

processors:
  - add_host_metadata:
      netinfo.enabled: true

logging.level: warning

http.enabled: true
http.port: 5069

https monitor config:

- type: http 
  id: my-kibana-http-monitor

  name: My Kibana HTTP Monitor

  schedule: '@every 10m' # every 5 seconds from start of beat

  hosts: ["https://kibana.hostname:5601"]

  ipv4: true
  mode: any

  ssl.enabled: true
  ssl:
    certificate_authorities: ["/foo/elastic-configs/heartbeat/elasticsearch-ca.pem"]
    verification_mode: "none"

And the only change required to break/fix the errors (which of course totally block that monitor from working, though naturally the server ping module that doesn't require any kind of ssl continues to work through) is flopping between:

certificate_authorities: ["/foo/elastic-configs/heartbeat/elasticsearch-ca.pem"]

and
certificate_authorities: ["${path.config}/elasticsearch-ca.pem"]

in the https monitor config. The variable expansion works properly in the main heartbeat config.

[DPattee]

This problem repros on multiple Mac machines and I think one of my raspberry pi's but I'd need to double check that one