# Filebeats - Sending JSON output to Compacted Kafka Topic

**URL:** <https://discuss.elastic.co/t/filebeats-sending-json-output-to-compacted-kafka-topic/188658>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [July 3, 2019, 9:00am UTC](https://discuss.elastic.co/t/filebeats-sending-json-output-to-compacted-kafka-topic/188658 "2019-07-03T09:00:56Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![vinshas1](https://avatars.discourse-cdn.com/v4/letter/v/3ec8ea/32.png) [@vinshas1](https://discuss.elastic.co/u/vinshas1)\
**Post date:** [July 3, 2019, 9:00am UTC](https://discuss.elastic.co/t/filebeats-sending-json-output-to-compacted-kafka-topic/188658/1 "2019-07-03T09:00:56Z")

</div>

I am able to send JSON output to Kafka non-compacted Topic but unable to send it to compacted topic. I know we need a keyed message to send it to compacted topic and I tried with 'key' in output.kafka and also tried json.message\_key: & json.keys\_under\_root: true options. May be I am not doing it correctly. Please help me in sending JSON output to compacted Kafka Topic using Filebeats.

**Error Message from Filebeats** :  
_Message contents does not match its CRC_

**Filebeats.yml**  
#=========================== Filebeat inputs =============================

filebeat.inputs:

- type: log  
enabled: true
# Update the path below to contain the log files you want to collect

# Paths that should be crawled and fetched. Glob based paths.
paths:  
#- /mnt/\*.json  
json.message\_key: message  
json.keys\_under\_root: true

processors:

- drop\_fields:  
fields: ["@metadata", "prospector","input","beat","host","source","offset","log"]
- decode\_json\_fields:  
fields: ["message"]

#-------------------------- kafka output ------------------------------  
output.kafka:

# Array of hosts to connect to.

hosts: ["xxxxx"]  
#topic: 'inventory'  
key: '%{[message.HostName]}'  
topic: 'inventory'

version: '2.0'  
compression: gzip  
max\_message\_bytes: 1000000  
required\_acks: 1

_ **Error Log from Filebeats** _:  
2019-07-03T03:25:31.710-0500 ERROR kafka/client.go:231 Kafka (topic=inventory): dropping invalid message

2019-07-03T03:25:31.710-0500 ERROR kafka/client.go:231 Kafka (topic=inventory): dropping invalid message

2019-07-03T03:25:31.710-0500 ERROR kafka/client.go:231 Kafka (topic=inventory): dropping invalid message

2019-07-03T03:25:31.711-0500 ERROR kafka/client.go:231 Kafka (topic=inventory): dropping invalid message

2019-07-03T03:25:31.711-0500 INFO kafka/log.go:53 producer/leader/inventory/1 state change to [retrying-3]

2019-07-03T03:25:31.711-0500 INFO kafka/log.go:53 producer/leader/inventory/1 abandoning broker 1027

2019-07-03T03:25:31.711-0500 INFO kafka/log.go:53 producer/broker/1027 state change to [closed] on inventory/1

2019-07-03T03:25:31.711-0500 INFO kafka/log.go:53 producer/broker/1027 shut down

2019-07-03T03:25:31.711-0500 WARN kafka/log.go:51 producer/broker/1026 state change to [retrying] on inventory/0 because kafka server: Message contents does not match its CRC.

---

<div class="post-metadata">

**Author:** ![vinshas1](https://avatars.discourse-cdn.com/v4/letter/v/3ec8ea/32.png) [@vinshas1](https://discuss.elastic.co/u/vinshas1)\
**Post date:** [July 5, 2019, 7:30am UTC](https://discuss.elastic.co/t/filebeats-sending-json-output-to-compacted-kafka-topic/188658/2 "2019-07-05T07:30:37Z")

</div>

Figured out myself. Thank You!!

---

<div class="post-metadata">

**Author:** ![vinshas1](https://avatars.discourse-cdn.com/v4/letter/v/3ec8ea/32.png) [@vinshas1](https://discuss.elastic.co/u/vinshas1)\
**Post date:** [July 5, 2019, 7:33am UTC](https://discuss.elastic.co/t/filebeats-sending-json-output-to-compacted-kafka-topic/188658/3 "2019-07-05T07:33:50Z")

</div>

Solution:

> You don't need json.message\_key so commented it out.  
> Change the Key Name to name of the key you get in the event.

#json.message\_key: message  
json.keys\_under\_root: true

key: '%{[Computer Name]}'

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 2, 2019, 7:33am UTC](https://discuss.elastic.co/t/filebeats-sending-json-output-to-compacted-kafka-topic/188658/4 "2019-08-02T07:33:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.