Filebeat sends garbled messages to kafka

[wangyanzhao]

use filebeat-6.1.2 to collect logs to kafka, but part of logs in kafka are Garbled.

filebeat.yml

path.data: ${FILEBEAT_HOME}/data
path.logs: ${FILEBEAT_HOME}/logs

filebeat.config.prospectors:
  enabled: true
  path: ${FILEBEAT_HOME}/config/*/*.yml
  reload.enabled: true
  reload.period: 6s

output.kafka:
  hosts: [...]
  topic: '%{[fields.kafka_topic]}'
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: snappy
  max_message_bytes: 1048576
  bulk_max_size: 2048
  codec.format:
    string: '%{[message]}'

max_procs: 2

logging.level: info
logging.to_files: true
logging.to_syslog: false
logging.files.rotateeverybytes: 524288000
logging.files:
  path: ${FILEBEAT_HOME}/logs
  name: filebeat.log
  keepfiles: 20
  permissions: 0644

prospectors.yml

- type: log
  paths:
  - /channel/dt/data/*
  fields:
    kafka_topic: ...
  scan_frequency: 10s
  harvester_buffer_size: 10485760
  multiline: null

logs sample

logs in kafka sample

[adrisr (Adrian Serrano)]

This is interesting.

What version of Kafka are you using?

Do you get any errors when this messages are processed in Logstash or Elasticsearch?

Are you seeing this error from all kinds of log files or just some particular logs?

[wangyanzhao]

1. kafka version is 0.11.0.2

2. There is no error in filebeat agent, we use kafka-connect to sink logs to hdfs, then use HIVE to processe logs.
   This is the HIVE error log.
   

3. Just some particular logs, this error has appeared three times. This time, the Garbled log was sent multiple times (at least once).
   

   

[adrisr (Adrian Serrano)]

I suggest you run filebeat with -d publish. This will allow you to see the events as they are published to Kafka.

Also you can use Wireshark with Kafka support to check if the logs from filebeat to Kafka are corrupt.

[wangyanzhao]

This problem has appeared again today. I am ready to run filebeat with -d publish, I hope to capture some useful information when the next problem reoccurs.

[steffens (Steffen Siering)]

Filebeat encodes JSON events into UTF-8. The exception is about decoding UTF-32. This can't work. The reader must use UTF-8.