Filebeat for binary files?

Elastic Stack Beats
1

Hello Folks,

I am new to whole Logstash/ELK. But I am wondering if I can use filebeat for streaming binary data to network ?
Here is what I am trying to do.

Multiple processes (in hundreds) have lib in them to encode data and write to corresponding "logs files"(binary encoded files to save space) of the processes. If we want to read them "log files" in human readable format we can to by running it them a decoder I have.

Now what I want to do is stream the "log files" to a remote location via filebeat and run my decoder there. So is this possible via logstash or filebeat ?

This is my firstpost and any help is much appreciated. Thanks.

P.S. I can run my decoder where the "log files" are being generated and feed the output location to filebeat but It kills both my cpu and network. So i want to stream the encoded files incrementally and use my decoder on the cloud.

2

Filebeat is designed to process text based logs line by line, and as far as I know it is not able to handle binary logs.

3

filebeat currently tails text files only looking for newlines (even removing newlin characters). You don't want to run this on binary files as a random sequence of '\r\n' in binary and just '\n' in binary you wont be able distinguish in your decoder (if any '\n' is missing you will never see your content).

There have been ideas to make filebeat prospectors more plugin-like for improved extensiblity, but this is not possible to date.

4

Thanks Steffen and Christian,

What do you guys suggest I do. Now I am planning on getting to write my own plugin for logstash or are there any plugins that do this already ? I'll post the same in logstash forum.

5

Binary encoded files... is a very very generic statement. Dealing with binary (and sometimes ascii ) content one needs specialised parsers being able to extract content. Dealing with text based log files is fortunately relatively straight forward as using newlines to separate log entries is a very commonly accepted habit.

You can check logstash input/filter/codec plugins for any support.

6

Hi Christian,

I am facing a similar problem: sending hexdump file from client running filebeat to server running logstash.
Since this was discussed almost a year ago, is there any plugin available now to filebeat to send hexdump / binary files?

Also is it possible to transfer files from client to server as is, that is without filebeat interpreting the content of file?

Thanks & Best wishes,
Veera

7

This is currently not possible.

8

Is there any support now for binary files from Filebeat ?

9

No, not that I am aware of. What kind of binary files are you looking to process?

10

Is there a way/option to "exclude" non-string data? I have log files which provides failing data records where sometime the SOH (x01) STX (x02) and ETX (x03) is part of the reported failing data.