Filebeat kafka ssl output

Elastic Stack Beats
#1

Trying to set up ssl kafka output in filebeat. reading a log file and sending to kafka.

non-ssl works fine. for ssl i am trying the below

#---- kafka output

output.kafka:
 # initial brokers for reading cluster metadata 
hosts: ["host1:9093", 
"host2:9093", "host3:9093"
ssl.certificate: /test/ssl/monitoring.crt
ssl.certificate_key: /test/ssl/monitoring.key
ssl.certificate_authorities: ["/test/ssl/digicertroot.cer"]

# message topic selection + partitioning
topic: 'filebeattest'
partition.round_robin:
reachable_only: false
required_acks: 1
compression: gzip
max_message_bytes: 1000000

I get the below error
ERROR instance/beat.go:958 Exiting: error initializing publisher: key file not
configured accessing 'output.kafka.ssl' (source:'filebeat.yml')
Exiting: error initializing publisher: key file not configured accessing
'output.kafka.ssl' (source:'filebeat.yml')

#2

Hi @madhan_dhanikachalam Welcome to the community.

First please make sure you format your yaml correctly.

You have several syntax issues.

hosts: ["host1:9093", 
"host2:9093", "host3:9093" <!---- Missing Closing Bracket and should be on the same line

Please review the SSL configuration Docs Here

Incorrect

ssl.certificate_key: /test/ssl/monitoring.key
ssl.certificate_authorities: ["/test/ssl/digicertroot.cer"]

should be

ssl.certificate.key: /test/ssl/monitoring.key
ssl.certificate.authorities: ["/test/ssl/digicertroot.cer"]

Fix those see what happens...

#3 
indent preformatted text by 4 spaces
output.kafka:
  hosts: ["host1:9093","host2:9093", "host3:9093"]
  ssl.certificate: /test/ssl/monitoring.crt
  ssl.certificate.key: /test/ssl/monitoring.key
  ssl.certificate.authorities: ["/test/ssl/digicertroot.cer"]
  topic: 'filebeattest'
  partition.round_robin:
  reachable_only: false
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000

formatting was good, maybe it was just me pasting on the forum. Changed the config field and now i am getting the below error

Exiting: error loading config file: required 'object', but found 'string' in field 
'ssl.certificate' (source:'filebeat.yml')
#4

turns out

ssl.certificate.key: 
ssl.certificate.authorities:

should be

ssl.key:
ssl.authorities:

worked with this change.

1 Like
#5

Yep apologies cut n pasta error my side