Issue with filebeat kafka output lz4 compression


(Prabin Meitei) #1

I am trying to setup filebeat (6.2) with kafka output using lz4 compression, but always getting error as
ERR Invalid kafka configuration: kafka: invalid configuration (lz4 compression requires Version >= V0_10_0_0)

My output configuration is:
output.kafka:
hosts: ["kafka1:10092","kafka2:10092","kafka3:10092"]
topic: '%{[fields.log_topic]}'
partition.round_robin:
reachable_only: false

required_acks: 1
compression: lz4
max_message_bytes: 1000000
version: 0.11.0.0

Any insight on what might be wrong would be very helpful

Thanks.


(Steffen Siering) #2

Might be a bug. What happens if you set version to 0.10?


(Prabin Meitei) #3

Got the same error. Another surprising thing, possibly another bug, is that if I specify the version as

version: 0.10

I got the error as

Exiting: error initializing publisher: unknown/unsupported kafka version '0.1' accessing 'output.kafka' (source:'filebeat.yml')

specifying version as (with quotes)

version: "0.10"

gives the original error

Invalid kafka configuration: kafka: invalid configuration (lz4 compression requires Version >= V0_10_0_0)


(Steffen Siering) #4

This error is to be expected:

Exiting: error initializing publisher: unknown/unsupported kafka version '0.1' accessing 'output.kafka' (source:'filebeat.yml')

When using 1.10 without quotes, the YAML parser/spec will interpret the input as floating point value :slight_smile:

With setting the version to '0.10', you shouldn't get this error message. This is clearly a bug.

I see you've opened: https://github.com/elastic/beats/issues/6414
Thanks!


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.