Help with Filebeat for Windows

[jnpetty (Jason)]

I am trying to test out filebeat on a Windows server. Currently I am echoing into a test file that filebeat is monitoring and then output to another file to see what the output looks like. I am writing to the test file with the following command:

echo "Does this work?" >> .\Logtest.log

In the output file the Message portion looks like:

"message":"\ufffd\ufffdD\u0000o\u0000e\u0000s\u0000 \u0000t\u0000h\u0000i\u0000s\u0000 \u0000w\u0000o\u0000r\u0000k\u0000?\u0000\r\u0000",

Thoughts on whats going on?

[andrewkroh]

You probably need to set the proper file encoding. See https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#_encoding

filebeat:
  prospectors:
    - paths:
        - c:\programdata\myapp\logs\*
      # Configure the file encoding for reading files with international characters
      # following the W3C recommendation for HTML5 (http://www.w3.org/TR/encoding).
      # Some sample encodings:
      #   plain, utf-8, utf-16be-bom, utf-16be, utf-16le, big5, gb18030, gbk,
      #    hz-gb-2312, euc-kr, euc-jp, iso-2022-jp, shift-jis, ...
      encoding: utf-8

[jnpetty (Jason)]

No dice, I tried both plain and utf-8 encoding, with no luck

Plain resulted in
"message":"��t\u0000e\u0000s\u0000t\u0000n\u0000o\u0000t\u0000i\u0000c\u0000e\u0000\r\u0000",

and utf-8 resulted in
"message":"\u0000t\u0000e\u0000s\u0000t\u0000n\u0000o\u0000t\u0000i\u0000c\u0000e\u0000\r\u0000",

I also just tried manually typing in data and saving and it was the same result.

[ruflin]

What editor are you using to type in the data?

[steffens (Steffen Siering)]

can you open/show the file in a hex editor? the \u0000 is the NULL-character for example. It's literaly a byte having value 0 (normally used to indicate end of C-strings).

If I remember correctly \ufffd stands for 'invalid.

Given you have NULL-character followed by a character I'd assume this being UTF-16 with some BOM sequence at beginning of file. Normally windows uses little endian based utf-16 encoding. Try encoding like utf-16le or utf-16le-bom .

[jnpetty (Jason)]

Thanks everyone for the help, I was able to get it working by changing the encoding to utf-16