I'm trying to integrate filebeat with Kafka with SSL Handshake. The certificates are obtained from vault and they are valid for only 7 days. A different mechanism is applied to get the certificates renewed with a new private key every 2 days so that the service will have zero down time (ZDT) and it reloads the certificates before they expire.
I've done the following configuration which monitors the certificates file path and reloads them when they've been changed.
filebeat.config.inputs:
enabled: true
path: /usr/share/filebeat/reload-configs/*.yml
reload.enabled: true
reload.period: 10s
output.kafka:
hosts: '${KAFKA_HOSTS}'
ssl.certificate: '${CERTS_PATH}/filebeat.pki.crt'
ssl.key: '${CERTS_PATH}/filebeat.pki.key'
ssl.authorities: ['${CERTS_PATH}/root_ca.pem']
topic: '${KAFKA_TOPIC}'
codec.format:
string: '{"timestamp": "%{[@timestamp]}", "message": %{[message]}, "host": %{[host]}}'
close_inactive: 10m
required_acks: 1
partition.round_robin:
reachable_only: false
keep-alive: 30000ms
kafka_filebeat_reload_configs.yml
- type: filestream
id: pki-crt
paths:
- ${CERTS_PATH}/filebeat.pki.crt
scan_frequency: 10s
- type: filestream
id: pki-key
paths:
- ${CERTS_PATH}/filebeat.pki.key
scan_frequency: 10s
There are couple of problems with this approach:
- It's outputting the file contents to Kafka topic
- It only reloads the files if number of lines are increased in these files (treating them as logs/filestream)
Is there any cleaner approach to reload the certificates without having to restart the filebeat process?
References: