Build your own Beat in Go

[adrianfusco (Adrian)]

I needed to send messages to Logstash using Beat from a little app I'm developing in Go.
I've seen different ways to do it without doing calling services as filebeat to not create a dependency.

I was taking a look to the next blog and it seems to be amazing: Build your own beat

The thing is that is using an old version from 2016: 5.1 and also used an old Go version.
I've seen something similar done in Python: pylogbeat · PyPI

Does anyone know if there's something similar made in Go? Maybe I can take advantage of the module and not depend of services in the machine as I mentioned.

Thank you very much.

[jsoriano (Jaime Soriano)]

Hey @adrianfusco,

Building your our beats is not supported anymore. Currently it is prefered to build your own integrations to monitor external services, or use APM to instrument your own applications.

What kind of messages are you trying to send? Is this operational data like logs or metrics?

If you are trying to send logs, metrics or traces from your application I would suggest you to instrument it using APM, take a look to this blog post about this: How to instrument your Go application with the Elastic APM Go agent | Elastic Blog

If you are trying to send some other kind of data, I would recommend to use the generic Elasticsearch client that you can find here: GitHub - elastic/go-elasticsearch: The official Go client for Elasticsearch

With any of these options you wouldn't need to run any other service on your machine. With APM your application would be sending the data to an APM server that could be running in a different server, and with the Elasticsearch client your application would be sending data directly to Elasticsearch.

[adrianfusco (Adrian)]

Thank you very much for your answer, @jsoriano

I have a little service running that decompress GZ files. As filebeat can't do it yet and it's in progress,

I wanted the content of these decompressed files messages to Logstash and receive them using the beats plugin.

Something similar to GitHub - brxie/PyLumberjack: Lumberjack protocol for python or GitHub - eht16/pylogbeat: Simple, incomplete implementation of the Beats protocol used by Elastic Beats and Logstash but using Go.

Idk if there's something already done or if I can use some module of the actual beats project.

Thank you very much again.

[adrianfusco (Adrian)]

I've seen now this project: GitHub - elastic/go-lumber: Go based lumberjack client and server implementation. I'll take a look.

[jsoriano (Jaime Soriano)]

Ah ok, so you are facing this verly-long standing issue https://github.com/elastic/beats/issues/637

If you are working on solving this issue, it'd be great if you could contribute it in filebeat, as a new input, or as an option in the filestream one.
The advantage of doing it as filebeat input is that you could reuse other functionality, as the queues or the outputs.

In the issue linked above you can find some of the challenges to support this feature.