How to create a new Beats output?

Elastic Stack Beats
1

Is there a standard interface/framework to create a new beats output. To be specific, we want to create a new output using which beats can send data to IBM Bluemix (logmet service).

Currently we have taken logstash ouput, changed it to add Bluemix authentication and send data. But this would mean we compile our own version of libbeat and use which we would like to avoid.

Is there a way one could write a new output and plugin with beats?

2

If you are thinking about creating an output, check out this comment here: https://github.com/elastic/beats/pull/1525#issuecomment-217651768 It is possible to create an output in a separate repository and only add it on compile time.

UPDATE: I removed the comment part about the generator as this is packetbeat protocol related. Sorry, too early in the morning ...

The above will still not prevent that you have to compile your own version of the beat, but it gets much simpler as you don't have to maintain a full fork of the beats repository.

Ping us if you hit some roadbloacks.

3

@ruflin, thank you for the input. Does this mean, I clone filebeat and libbeat and then modify filebeat's main.go and libbeat/publisher/publish.go and build ?

4

Yes, but I think you only need to modify the main.go and need to modify the publish.go. Your own output will be in a separate repo / package that you will refer to in the main.go file.

5

There is no need to fork filebeat or libbeat. The output and the actual main.go can each reside in their own repository.

e.g. let's assume repository myfilebeat having only this main.go:
package main

import (
    "os"

        _ "github.com/myuser/mybeatsoutputs/myoutput"

    "github.com/elastic/beats/filebeat/beater"
    "github.com/elastic/beats/libbeat/beat"
)

func main() {
    if err := beat.Run("myfilebeat, "", beater.New()); err != nil {
        os.Exit(1)
    }
}

In this example the output is available in repository github.com/myuser/mybeatsoutputs. With internal plugin achritecture of outputs, the output will be immediately available. Filebeat itself here acts like the 'framework' to use.

How you structure your repositories is all up to you. You can also put all custom beats + outputs into one repository and use the import trick on any beat, as every beat available can act as 'framework'.

6

Thank you. But how will I build this ? My need is to use this new output from filebeat.

7

and my output is an extended version of logstash output

8

@ruflin, This is what I did to create a new output, build and test with filebeat

a) copied logstash output to my repo (ex: github.com/ageetha/logmet)
b) changed package name and references from logstash to logmet ( I did
this as my output is a modified version of logstash output)
c) cloned this repo @ /root/src/github.com/
d) cloned beats 1.3 @ /root/src/github.com
e) modified /root/src/github.com/elastic/beats/filebeat/main.go to include imports to my repo
f)
modified /root/src/gitbub.com/elastic/beats/libbeat/outputs/outputs.go (
inclded two new parameters I introduced in yml file )
g) built filebeat

is there a way to introduce output specific yml parameters without
having to modify libbeat/outputs/outputs.go. I would like to keep all my
output specific code and config within my repo

9

There is no need to clone or copy filebeat. See my post from 7 days ago.

You might have 2 repositories for example:

  1. github.com/ageetha/logmet holding the new output plugin
  2. github.com/ageetha/filebeat-logmet holding your custom filebeat with logmet being included.

In github.com/ageetha/filebeat-logmet you have only one file named main.go with content:

import (
    "os"

    _ "github.com/ageetha/logmet"

    "github.com/elastic/beats/filebeat/beater"
    "github.com/elastic/beats/libbeat/beat"
)

Name := "filebeat-logmet"

func main() {
    if err := beat.Run(Name, "", beater.New()); err != nil {
        os.Exit(1)
    }
}

No run go build in $GOPATH/src/github.com/ageetha/filebeat-logmet directory and you will get your filebeat binary. Advantage of this solution is, you can more easily upgrade to new filebeat just by checking another branch from filebeat. Updating to filebeat 5.0 will require you to change main.go to:

import (
    "os"

        _ "github.com/myuser/mybeatsoutputs/myoutput"

    "github.com/elastic/beats/filebeat/beater"
    "github.com/elastic/beats/libbeat/beat"
)

func main() {
    if err := beat.Run("myfilebeat, "", beater.New); err != nil {
        os.Exit(1)
    }
}

Beats are made this way, to reduce the amount of maintenance normally required when copying and modifying some other projects source code.

10

Thank you. This is helpful. I will try this.

11

This topic was automatically closed after 21 days. New replies are no longer allowed.