Previously discussed by others here: Embedding a custom beat in my application
So far libbeat has been intended to be used to produce standalone data shippers by using the beats framework to generate the project layout. I've used it to publish a few beats this way. The name implies it's a library, and it would be very useful as one, but it seems difficult to use it that way right now. What if you have an existing application or server that you wish to add data shipping to? Libbeats gives you flexibility of different outputs, buffering events for performance, metrics of output performance, and a familiar configuration, so it seems to fit the bill for that too, better than anything else I could find.
I integrated it into an application, and it seems like it should be easier to do and documented how to do so.
Here are some pain points I ran into:
- Starting it up is not clear. The standard framework does some sort of mysterious
beat.Run(yourInterfaceImplementation)thing, and I had to figure out how to string together different library components to work similarly and integrate with my application. - Dependency issues with
dep: I had to change a few dependency revisions manually to match what libbeat has in https://github.com/elastic/beats/blob/master/vendor/vendor.json - Dependency on CGO? - gosigar dependency seems to require CGO, but not necessarily? I know some beats are built with CGO and some without. It's not clear to me how I can choose. Compiling with CGO_ENABLED=0 seems to still complain about GCC not being present. Any help with this would be appreciated.
- It would be nice to optionally turn off (and not compile in) features like templates, dashboards, system information collection, etc.
Here is what I'm using at the moment: https://github.com/dustin-decker/threatseer/blob/v2/server/engines/shipper/beat.go
Here is something to work from if I take that down or change it substantially:
package shipper
import (
"fmt"
"log"
"time"
"github.com/dustin-decker/threatseer/server/event"
"github.com/elastic/beats/libbeat/beat"
"github.com/elastic/beats/libbeat/cmd/instance"
"github.com/elastic/beats/libbeat/common"
)
// Shipper makes it compatible flow pipeline
type Shipper struct {
done chan struct{}
config Config
client beat.Client
}
// Start is the entrypoint from the flow pipeline
func (s *Shipper) Start(in chan event.Event) {
for {
e := <-in
evnt := beat.Event{
Timestamp: time.Now(),
Fields: common.MapStr{
"event": e.Event,
"indicators": e.Indicators,
"src_ip": e.ClientAddr,
},
}
log.Print("trying to publish...")
// goes to output
s.client.Publish(evnt)
log.Print("api", "event sent: %v", evnt)
}
}
// NewShipperEngine is the entrypoint for the datashipper
func NewShipperEngine() Shipper {
bt, err := instance.NewBeat("threatseer", "", "")
if err != nil {
log.Fatal("could not instantiate beat, got: ", err)
}
err = bt.Setup(newShipper, false, false, false)
if err != nil {
log.Fatal("error setting up the shipper, got: ", err)
}
client, err := bt.Publisher.Connect()
if err != nil {
log.Fatal("error connecting to shipper output, got: ", err)
}
return Shipper{
done: make(chan struct{}),
config: DefaultConfig,
client: client,
}
}
// just here to satisfty instance.Beat.Setup
// you can load in custom configs here as usual
func newShipper(b *beat.Beat, cfg *common.Config) (beat.Beater, error) {
config := DefaultConfig
if err := cfg.Unpack(&config); err != nil {
return nil, fmt.Errorf("Error reading config file: %v", err)
}
bt := &Shipper{}
return bt, nil
}
// Run starts the beater daemon
// This is only here to satisfy the interface
func (s *Shipper) Run(b *beat.Beat) error {
ticker := time.NewTicker(s.config.Interval)
for {
select {
case <-s.done:
log.Print("recieved done signal, shutting down event shipper")
return nil
case <-ticker.C:
}
}
}
// Stop gets called when libbeat gets a SIGTERM.
func (s *Shipper) Stop() {
err := s.client.Close()
if err != nil {
log.Print("stopping the beat client failed because of: ", err)
}
close(s.done)
}
Thanks for the hard work on libbeat. It's good tech, and I hope implementing it will be more flexible in the future because a lot of people would find it useful.
Mentioning @pkaeding in case you're still interested and around.