Filebeat converting special character into unicode value when loading into kafka

Elastic Stack Beats
1

Hello Everyone,

I am trying to load data into kafka topic through filebeat. The major issue I am facing is, filebeat is converting special character (&) into Unicode value (\u0026) when loading into kafka topic output.
My log data has the field "department": "E&C". When this field is loaded into kafka from filebeat, it is looking as E\u0026C. I am using latest version of filebeat 7.13.3.

How I can resolve this issue?. I have tried lot of different possibilities but nothing works out. please advise

This is my .yml file:

filebeat.inputs:
-type: log
paths:
- C:\Users\MYNAME\Downloads\abc.log
multiline.type: pattern
multiline.pattern: '###'
multiline.negate: true
multiline.match: after

output.kafka:
enabled: true
hosts: [localhost:9092]
topic: "My_topic"
codec.json:
pretty: true
escape_html: false

Note: I have tried with and without "escape_html" option. But in both case, Unicode conversion is happening.

Please anyone help me in resolving this issue. It would be really great help for me.

Thanks in advance

2

Hi!

It seems to be similar to Why Filebeat returns Unicode character code instead of XML tag symbol under message, it is showing like \u003c and \u003e - #8 by manjsr. I wonder if there was a fix for the other issue or not. Could you check how the events are logged in your console and see if the strings are escaped there or not? If we verify that escape_html doesn't work we can open a GH for this so as to have the team work on fix.

C.

3

Hi Chris Mark,

Thanks for your response. Yes, I have tried pushing the result into console as well as tried using other output plug-in like writing result to a file, In both case, I could see '&' in the result ('&' is not converted to '\u0026'). But I am not sure why this conversion happening when loading the data into kafka using output.kafka plug in. I couldn't find any solution for this problem.

Kindly help.

4

Could anyone please help me on resolving this issue.. is this a problem that yet to be fixed in filebeat-kafka output plugin?