Filebeat sends garbled messages \u0000

[Alexik]

Filebeat sends garbled messages \u0000

Hi guys,

I'm having problem with Filebeat. From time to time I'm receiving an event which looks like this:

{
"_index": "uat1-access-ws",
"_type": "log",
"_id": "AVj4Nau7dy5nH_giebKm",
"_score": null,
"_source": {
"duration": null,
"target_index": "ws",
"@timestamp": "2016-12-13T12:43:32.314Z",
"offset": 5226593,
"@version": "1",
"input_type": "log",
"beat": {
"hostname": "xxxxx",
"name": "xxxxx",
"version": "5.1.1"
},
"host": "xxxxx",
"source": "xxxxx/ws_access.log",
"message": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000_91" CT="text/xml" CE="-" PID="_ib-20161213134330-682154" WS="GetAddressSuggestions"",
"type": "log",
"tags": [
"beats_input_codec_plain_applied",
"_grokparsefailure"
]

In Filebeat(5.1.1) config there is nothing fancy. I have tried to use charset utf-8/us-ascii(it's used in our logfiles) but no change. I've located particular events in logfiles and it looks exactly same as events above/belove in whole logfile.

Any idea what to do with this ?

Thanks in advance

[steffens (Steffen Siering)]

Are you using a network share? I've seen this happening in case of some network/output buffering writting the second part before the first part being buffered. In this case the OS will fill up the missing write with zeros.

The \u0000 is not 'garbage', but a buffer full of bytes of value 0. With zero being no readable ASCII-character, the 0 will be encoded as \u0000.

[Alexik]

Thanks for your reply.
Filebeat runs on server with logs and send them via beat protocol to listening logstash on particular port. Quite normal scenario.
So I think I will try to dig deeper in network stack and see what I can find. I will try to keep this threat up-to-date.

[ruflin]

Also have a look at Filebeat 5.0 with multiline, split event data to two events where this \u0000 also showed up.

[steffens (Steffen Siering)]

who is producing the logs? Is the disk remote or local?