Filebeats - Sending JSON output to Compacted Kafka Topic

Elastic Stack Beats
1

I am able to send JSON output to Kafka non-compacted Topic but unable to send it to compacted topic. I know we need a keyed message to send it to compacted topic and I tried with 'key' in output.kafka and also tried json.message_key: & json.keys_under_root: true options. May be I am not doing it correctly. Please help me in sending JSON output to compacted Kafka Topic using Filebeats.

Error Message from Filebeats:
Message contents does not match its CRC

Filebeats.yml
#=========================== Filebeat inputs =============================

filebeat.inputs:

  • type: log
    enabled: true

    Update the path below to contain the log files you want to collect

    Paths that should be crawled and fetched. Glob based paths.

    paths:
    #- /mnt/*.json
    json.message_key: message
    json.keys_under_root: true

processors:

  • drop_fields:
    fields: ["@metadata", "prospector","input","beat","host","source","offset","log"]
  • decode_json_fields:
    fields: ["message"]

#-------------------------- kafka output ------------------------------
output.kafka:

Array of hosts to connect to.

hosts: ["xxxxx"]
#topic: 'inventory'
key: '%{[message.HostName]}'
topic: 'inventory'

version: '2.0'
compression: gzip
max_message_bytes: 1000000
required_acks: 1

Error Log from Filebeats:
2019-07-03T03:25:31.710-0500 ERROR kafka/client.go:231 Kafka (topic=inventory): dropping invalid message

2019-07-03T03:25:31.710-0500 ERROR kafka/client.go:231 Kafka (topic=inventory): dropping invalid message

2019-07-03T03:25:31.710-0500 ERROR kafka/client.go:231 Kafka (topic=inventory): dropping invalid message

2019-07-03T03:25:31.711-0500 ERROR kafka/client.go:231 Kafka (topic=inventory): dropping invalid message

2019-07-03T03:25:31.711-0500 INFO kafka/log.go:53 producer/leader/inventory/1 state change to [retrying-3]

2019-07-03T03:25:31.711-0500 INFO kafka/log.go:53 producer/leader/inventory/1 abandoning broker 1027

2019-07-03T03:25:31.711-0500 INFO kafka/log.go:53 producer/broker/1027 state change to [closed] on inventory/1

2019-07-03T03:25:31.711-0500 INFO kafka/log.go:53 producer/broker/1027 shut down

2019-07-03T03:25:31.711-0500 WARN kafka/log.go:51 producer/broker/1026 state change to [retrying] on inventory/0 because kafka server: Message contents does not match its CRC.

2

Figured out myself. Thank You!!

3

Solution:

You don't need json.message_key so commented it out.
Change the Key Name to name of the key you get in the event.

#json.message_key: message
json.keys_under_root: true

key: '%{[Computer Name]}'

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.