Kafka formatting output

Elastic Stack Beats
1

Using 5.2.2, trying to send just my log message to Kafka by defining the codec in filebeat.yml as shown below:

filebeat:
  prospectors: []
  registry_file: "/var/lib/filebeat/registry"
  config_dir: "/etc/filebeat/conf.d"
output:
  kafka:
enable: true
hosts:
- myhost:9092
topic: mytopic
required_acks: 1
compression: gzip
max_message_bytes: 1000000
timeout: 90s
broker_timeout: 60s
codec:
  format:
    string: "%{[message]}"

Using the above, I still get all the metadata sent to Kafka, not just the message. I suspect my yml isn't formatted correctly since earlier posts in this group say support for this was available as of 5.2.0. I'd appreciate any help showing me where I'm going wrong. Thanks!

2

Apologies. I think I was too impatient for this feature to be released. Using the latest snapshot build here https://beats-nightlies.s3.amazonaws.com/filebeat/filebeat-6.0.0-alpha1-SNAPSHOT-linux-x86_64.tar.gz , I'm getting the exact behavior I was hoping for. Nice job devs, I'll pay more attention next time. Sorry for any confusion.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.