# Filebeat sends garbled messages to kafka

**URL:** <https://discuss.elastic.co/t/filebeat-sends-garbled-messages-to-kafka/129277>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 24, 2018, 9:47am UTC](https://discuss.elastic.co/t/filebeat-sends-garbled-messages-to-kafka/129277 "2018-04-24T09:47:06Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![wangyanzhao](https://avatars.discourse-cdn.com/v4/letter/w/e19b73/32.png) [@wangyanzhao](https://discuss.elastic.co/u/wangyanzhao)\
**Post date:** [April 24, 2018, 9:47am UTC](https://discuss.elastic.co/t/filebeat-sends-garbled-messages-to-kafka/129277/1 "2018-04-24T09:47:06Z")

</div>

use filebeat-6.1.2 to collect logs to kafka, but part of logs in kafka are Garbled.

filebeat.yml

```

path.data: ${FILEBEAT_HOME}/data
path.logs: ${FILEBEAT_HOME}/logs

filebeat.config.prospectors:
  enabled: true
  path: ${FILEBEAT_HOME}/config/*/*.yml
  reload.enabled: true
  reload.period: 6s

output.kafka:
  hosts: [...]
  topic: '%{[fields.kafka_topic]}'
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: snappy
  max_message_bytes: 1048576
  bulk_max_size: 2048
  codec.format:
    string: '%{[message]}'

max_procs: 2

logging.level: info
logging.to_files: true
logging.to_syslog: false
logging.files.rotateeverybytes: 524288000
logging.files:
  path: ${FILEBEAT_HOME}/logs
  name: filebeat.log
  keepfiles: 20
  permissions: 0644

```

prospectors.yml

```

- type: log
  paths:
  - /channel/dt/data/*
  fields:
    kafka_topic: ...
  scan_frequency: 10s
  harvester_buffer_size: 10485760
  multiline: null

```

logs sample

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/0/c0e3000f0c3f1ac81ee8316fc976c885bfcdcc53.png)

logs in kafka sample

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/8/18c61c638d5079cec86dcc7aff691bdb9ec0d8e6.png)

---

<div class="post-metadata">

**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\
**Post date:** [April 24, 2018, 10:40am UTC](https://discuss.elastic.co/t/filebeat-sends-garbled-messages-to-kafka/129277/2 "2018-04-24T10:40:57Z")

</div>

This is interesting.

What version of Kafka are you using?

Do you get any errors when this messages are processed in Logstash or Elasticsearch?

Are you seeing this error from all kinds of log files or just some particular logs?

---

<div class="post-metadata">

**Author:** ![wangyanzhao](https://avatars.discourse-cdn.com/v4/letter/w/e19b73/32.png) [@wangyanzhao](https://discuss.elastic.co/u/wangyanzhao)\
**Post date:** [April 24, 2018, 11:35am UTC](https://discuss.elastic.co/t/filebeat-sends-garbled-messages-to-kafka/129277/3 "2018-04-24T11:35:10Z")

</div>

1. kafka version is 0.11.0.2

2. There is no error in filebeat agent, we use kafka-connect to sink logs to hdfs, then use HIVE to processe logs.  
This is the HIVE error log.  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/4/c47d05498509f32add9c2c83c50dbe9bec617043.png)

3. Just some particular logs, this error has appeared three times. This time, the Garbled log was sent multiple times (at least once).

---

<div class="post-metadata">

**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\
**Post date:** [April 25, 2018, 6:45am UTC](https://discuss.elastic.co/t/filebeat-sends-garbled-messages-to-kafka/129277/4 "2018-04-25T06:45:34Z")

</div>

I suggest you run filebeat with `-d publish`. This will allow you to see the events as they are published to Kafka.

Also you can use Wireshark with Kafka support to check if the logs from filebeat to Kafka are corrupt.

---

<div class="post-metadata">

**Author:** ![wangyanzhao](https://avatars.discourse-cdn.com/v4/letter/w/e19b73/32.png) [@wangyanzhao](https://discuss.elastic.co/u/wangyanzhao)\
**Post date:** [April 27, 2018, 11:45am UTC](https://discuss.elastic.co/t/filebeat-sends-garbled-messages-to-kafka/129277/5 "2018-04-27T11:45:04Z")

</div>

This problem has appeared again today. I am ready to run filebeat with -d publish, I hope to capture some useful information when the next problem reoccurs.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [May 2, 2018, 10:01am UTC](https://discuss.elastic.co/t/filebeat-sends-garbled-messages-to-kafka/129277/6 "2018-05-02T10:01:52Z")

</div>

Filebeat encodes JSON events into UTF-8. The exception is about decoding UTF-32. This can't work. The reader must use UTF-8.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 30, 2018, 10:01am UTC](https://discuss.elastic.co/t/filebeat-sends-garbled-messages-to-kafka/129277/7 "2018-05-30T10:01:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.