Unable to Reload certificates on filebeat container for Kafka

Elastic Stack Beats
1

I'm trying to integrate filebeat with Kafka with SSL Handshake. The certificates are obtained from vault and they are valid for only 7 days. A different mechanism is applied to get the certificates renewed with a new private key every 2 days so that the service will have zero down time (ZDT) and it reloads the certificates before they expire.

I've done the following configuration which monitors the certificates file path and reloads them when they've been changed.


filebeat.config.inputs:
  enabled: true
  path: /usr/share/filebeat/reload-configs/*.yml
  reload.enabled: true
  reload.period: 10s

output.kafka:
  hosts: '${KAFKA_HOSTS}'
  ssl.certificate: '${CERTS_PATH}/filebeat.pki.crt'
  ssl.key: '${CERTS_PATH}/filebeat.pki.key'
  ssl.authorities: ['${CERTS_PATH}/root_ca.pem']
  topic: '${KAFKA_TOPIC}'
  codec.format:
    string: '{"timestamp": "%{[@timestamp]}", "message": %{[message]}, "host": %{[host]}}'
  close_inactive: 10m
  required_acks: 1
  partition.round_robin:
    reachable_only: false
  keep-alive: 30000ms

kafka_filebeat_reload_configs.yml

- type: filestream
  id: pki-crt
  paths:
    - ${CERTS_PATH}/filebeat.pki.crt
  scan_frequency: 10s
- type: filestream
  id: pki-key
  paths:
    - ${CERTS_PATH}/filebeat.pki.key
  scan_frequency: 10s

There are couple of problems with this approach:

  1. It's outputting the file contents to Kafka topic
  2. It only reloads the files if number of lines are increased in these files (treating them as logs/filestream)

Is there any cleaner approach to reload the certificates without having to restart the filebeat process?

References:

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.