Is there any Beats solution for using a custom binary for decoding logs/metrics?

Hi all.

I would like to test whether an EBK (es, beats, kibana) stack could be setup to work in my situation. I am very new to Beats and only was introduced to it in the past workshop I recently attended.

We have a custom binary log that our app generates, which in turn requires a custom binary decoder to extract data from.

Is there anyway I can use File Beats or some sort of other Beats module that can execute the decoder and send output to ES?

Example syntax of decoder:

./binary_decoder -env dev -binary_log '/opt/data/app/binary_log_010203020302_binary.log'

Redirecting output to file populates something like:

dev.myApp.coreSlave1113.component.backend.rpc.GetSomeParams.count 458 1549078080
dev.myApp.coreSlave1113.component.backend.rpc.GetSomeParams.successcount 458 1549078080
dev.myApp.coreSlave1113.component.backend.rpc.GetSomeParams.failcount 0 1549078080
dev.myApp.coreSlave1113.component.backend.rpc.GetSomeParams.authfailcount 0 1549078080
dev.myApp.coreSlave1113.component.backend.rpc.GetSomeParams.totaltime 545294 1549078080

then formulate desired data structure via regex:

*.*.*.component.*.rpc.*.count
name="myApp_rpc_count"
myApp_environment="${1}"
myApp_service="${3}"
myApp_component="${7}"
myApp_rpc="${8}"

(The above two blocks is from the graphite_exporter that we currently have setup)

I would like to send the above to ES directly via Beats, rather than hack something additional together where I need to create some wrapper to extract files to some dir where file beats can read of off, and have to worry about implementing some mechanism where I then need clean the files outputted by this hack so that my partition doesn't run out of space etc.

I just found some pages on Logstash and a graphite plugin. I would also like to avoid having to setup Logstash as that would introduce yet another tech I need to setup and install.

Thank you in advance...

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.