Embedding libbeat in an application


(Dustin Decker) #1

Previously discussed by others here: Embedding a custom beat in my application

So far libbeat has been intended to be used to produce standalone data shippers by using the beats framework to generate the project layout. I've used it to publish a few beats this way. The name implies it's a library, and it would be very useful as one, but it seems difficult to use it that way right now. What if you have an existing application or server that you wish to add data shipping to? Libbeats gives you flexibility of different outputs, buffering events for performance, metrics of output performance, and a familiar configuration, so it seems to fit the bill for that too, better than anything else I could find.

I integrated it into an application, and it seems like it should be easier to do and documented how to do so.

Here are some pain points I ran into:

  1. Starting it up is not clear. The standard framework does some sort of mysterious beat.Run(yourInterfaceImplementation) thing, and I had to figure out how to string together different library components to work similarly and integrate with my application.
  2. Dependency issues with dep: I had to change a few dependency revisions manually to match what libbeat has in https://github.com/elastic/beats/blob/master/vendor/vendor.json
  3. Dependency on CGO? - gosigar dependency seems to require CGO, but not necessarily? I know some beats are built with CGO and some without. It's not clear to me how I can choose. Compiling with CGO_ENABLED=0 seems to still complain about GCC not being present. Any help with this would be appreciated.
  4. It would be nice to optionally turn off (and not compile in) features like templates, dashboards, system information collection, etc.

Here is what I'm using at the moment: https://github.com/dustin-decker/threatseer/blob/v2/server/engines/shipper/beat.go

Here is something to work from if I take that down or change it substantially:

package shipper

import (
	"fmt"
	"log"
	"time"

	"github.com/dustin-decker/threatseer/server/event"
	"github.com/elastic/beats/libbeat/beat"
	"github.com/elastic/beats/libbeat/cmd/instance"
	"github.com/elastic/beats/libbeat/common"
)

// Shipper makes it compatible flow pipeline
type Shipper struct {
	done   chan struct{}
	config Config
	client beat.Client
}

// Start is the entrypoint from the flow pipeline
func (s *Shipper) Start(in chan event.Event) {
	for {
		e := <-in
		evnt := beat.Event{
			Timestamp: time.Now(),
			Fields: common.MapStr{
				"event":      e.Event,
				"indicators": e.Indicators,
				"src_ip":     e.ClientAddr,
			},
		}

		log.Print("trying to publish...")
		// goes to output
		s.client.Publish(evnt)
		log.Print("api", "event sent: %v", evnt)
	}
}

// NewShipperEngine is the entrypoint for the datashipper
func NewShipperEngine() Shipper {
	bt, err := instance.NewBeat("threatseer", "", "")
	if err != nil {
		log.Fatal("could not instantiate beat, got: ", err)
	}

	err = bt.Setup(newShipper, false, false, false)
	if err != nil {
		log.Fatal("error setting up the shipper, got: ", err)
	}

	client, err := bt.Publisher.Connect()
	if err != nil {
		log.Fatal("error connecting to shipper output, got: ", err)
	}

	return Shipper{
		done:   make(chan struct{}),
		config: DefaultConfig,
		client: client,
	}
}

// just here to satisfty instance.Beat.Setup
// you can load in custom configs here as usual
func newShipper(b *beat.Beat, cfg *common.Config) (beat.Beater, error) {
	config := DefaultConfig
	if err := cfg.Unpack(&config); err != nil {
		return nil, fmt.Errorf("Error reading config file: %v", err)
	}

	bt := &Shipper{}
	return bt, nil
}

// Run starts the beater daemon
// This is only here to satisfy the interface
func (s *Shipper) Run(b *beat.Beat) error {
	ticker := time.NewTicker(s.config.Interval)
	for {
		select {
		case <-s.done:
			log.Print("recieved done signal, shutting down event shipper")
			return nil
		case <-ticker.C:
		}
	}
}

// Stop gets called when libbeat gets a SIGTERM.
func (s *Shipper) Stop() {
	err := s.client.Close()
	if err != nil {
		log.Print("stopping the beat client failed because of: ", err)
	}
	close(s.done)
}

Thanks for the hard work on libbeat. It's good tech, and I hope implementing it will be more flexible in the future because a lot of people would find it useful.

Mentioning @pkaeding in case you're still interested and around.


(Dustin Decker) #2

Oops - could this be moved to Beats Developers? Feel free to delete this reply when it's moved.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.


(Pier-Hugues Pellerin) #4

@dustindecker Thanks for taking the time to report that. We are aware of some pitfalls to embed our lib in other application and we are currently working to improve that. The first issue I am looking at is to allow you to have finer controls on what libbeat can provide and reduce your compile size / time.

See this issue: https://github.com/elastic/beats/issues/7093


(Mark Walkom) #5

(Dustin Decker) #6

Thanks @pierhugues, I'll take a look at that issue.

I ended up abandoning the approach I posted above, and went the standard route of building the application around the beater interface. The reason was because of hidden behavior like not emitting logs to stdout unless -e flag was present, so it seemed more straightforward to just embrace the current recommended approach.