Filebeat kafka input message is incomplete and full of Unicode characters

Elastic Stack Beats
1

Hi Forum

We have a filebeat instance that uses the kafka input to read from a topic.
The instance is able to connect to the topic, but the output of the "message" key is incomplete, and full of Unicode characters see the below example.

"message": "\u0000\u0000\u0000\u0000\u0004Hfe4b2fd2-fd86-4b0e-a157-3ec3dc98f1b0\u0000\ufffd\u0001ebc0604a2673385a3b10b5a02ed4bbaf8c0135658efce5d7b3c4096a2c8f49f0Vsplit_ICC_head_orientation_90_90_mask_train\u0004{}\u0002\nOwner\u001cMyuser\u0000\ufffd\u0001129f6167512d54e6db7f32f2cf38319e10c5868bc49780208d24158b32b40f72\ufffd\u0001FA30_NIR__2021-11-18_15-21-51__Session_2.dmp_00000283_017d69b727_0.tiffVsplit_ICC_head_orientation_90_90_mask_train\u0006ICM${"pairs":"1 5053"}"

The source message have a "operation" key/value that is not shown in the message above so it is missing.

Any idea why the message content is full of the unicode characters?

Other kafka consumers show the message as a json string representation.

Thanks in advance

2

Hi Here
Just for future users reporting similar problem with filebeat and Kafka messages.
If the kafka message topic is not serialized as string or JSON filebeat does not know how to rendered.
So if you use custom serializers or AVRO you are out of luck with filebeat.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.