# How to create a new Beats output?

**URL:** <https://discuss.elastic.co/t/how-to-create-a-new-beats-output/61074>\
**Category:** Beats\
**Created:** [September 21, 2016, 4:06am UTC](https://discuss.elastic.co/t/how-to-create-a-new-beats-output/61074 "2016-09-21T04:06:59Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Geetha\_Adinarayan](https://avatars.discourse-cdn.com/v4/letter/g/c6cbf5/32.png) [@Geetha\_Adinarayan](https://discuss.elastic.co/u/Geetha_Adinarayan)\
**Post date:** [September 21, 2016, 4:06am UTC](https://discuss.elastic.co/t/how-to-create-a-new-beats-output/61074/1 "2016-09-21T04:06:59Z")

</div>

Is there a standard interface/framework to create a new beats output. To be specific, we want to create a new output using which beats can send data to IBM Bluemix (logmet service).

Currently we have taken logstash ouput, changed it to add Bluemix authentication and send data. But this would mean we compile our own version of libbeat and use which we would like to avoid.

Is there a way one could write a new output and plugin with beats?

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [September 21, 2016, 6:51am UTC](https://discuss.elastic.co/t/how-to-create-a-new-beats-output/61074/2 "2016-09-21T06:51:30Z")

</div>

If you are thinking about creating an output, check out this comment here: [https://github.com/elastic/beats/pull/1525#issuecomment-217651768](https://github.com/elastic/beats/pull/1525#issuecomment-217651768) It is possible to create an output in a separate repository and only add it on compile time.

UPDATE: I removed the comment part about the generator as this is packetbeat protocol related. Sorry, too early in the morning ...

The above will still not prevent that you have to compile your own version of the beat, but it gets much simpler as you don't have to maintain a full fork of the beats repository.

Ping us if you hit some roadbloacks.

---

<div class="post-metadata">

**Author:** ![Geetha\_Adinarayan](https://avatars.discourse-cdn.com/v4/letter/g/c6cbf5/32.png) [@Geetha\_Adinarayan](https://discuss.elastic.co/u/Geetha_Adinarayan)\
**Post date:** [September 21, 2016, 3:14pm UTC](https://discuss.elastic.co/t/how-to-create-a-new-beats-output/61074/3 "2016-09-21T15:14:40Z")

</div>

@ruflin, thank you for the input. Does this mean, I clone filebeat and libbeat and then modify filebeat's main.go and libbeat/publisher/publish.go and build ?

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [September 22, 2016, 7:58am UTC](https://discuss.elastic.co/t/how-to-create-a-new-beats-output/61074/4 "2016-09-22T07:58:05Z")

</div>

Yes, but I think you only need to modify the main.go and need to modify the `publish.go`. Your own output will be in a separate repo / package that you will refer to in the `main.go` file.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [September 22, 2016, 1:06pm UTC](https://discuss.elastic.co/t/how-to-create-a-new-beats-output/61074/5 "2016-09-22T13:06:16Z")

</div>

There is no need to fork filebeat or libbeat. The output and the actual `main.go` can each reside in their own repository.

e.g. let's assume repository `myfilebeat` having only this `main.go`:  
package main

```auto
import (
    "os"

        _ "github.com/myuser/mybeatsoutputs/myoutput"

    "github.com/elastic/beats/filebeat/beater"
    "github.com/elastic/beats/libbeat/beat"
)

func main() {
    if err := beat.Run("myfilebeat, "", beater.New()); err != nil {
        os.Exit(1)
    }
}

```

In this example the output is available in repository `github.com/myuser/mybeatsoutputs`. With internal plugin achritecture of outputs, the output will be immediately available. Filebeat itself here acts like the 'framework' to use.

How you structure your repositories is all up to you. You can also put all custom beats + outputs into one repository and use the import trick on any beat, as every beat available can act as 'framework'.

---

<div class="post-metadata">

**Author:** ![Geetha\_Adinarayan](https://avatars.discourse-cdn.com/v4/letter/g/c6cbf5/32.png) [@Geetha\_Adinarayan](https://discuss.elastic.co/u/Geetha_Adinarayan)\
**Post date:** [September 26, 2016, 6:24am UTC](https://discuss.elastic.co/t/how-to-create-a-new-beats-output/61074/6 "2016-09-26T06:24:23Z")

</div>

Thank you. But how will I build this ? My need is to use this new output from filebeat.

---

<div class="post-metadata">

**Author:** ![Geetha\_Adinarayan](https://avatars.discourse-cdn.com/v4/letter/g/c6cbf5/32.png) [@Geetha\_Adinarayan](https://discuss.elastic.co/u/Geetha_Adinarayan)\
**Post date:** [September 26, 2016, 10:59am UTC](https://discuss.elastic.co/t/how-to-create-a-new-beats-output/61074/7 "2016-09-26T10:59:12Z")

</div>

and my output is an extended version of logstash output

---

<div class="post-metadata">

**Author:** ![Geetha\_Adinarayan](https://avatars.discourse-cdn.com/v4/letter/g/c6cbf5/32.png) [@Geetha\_Adinarayan](https://discuss.elastic.co/u/Geetha_Adinarayan)\
**Post date:** [September 26, 2016, 11:00am UTC](https://discuss.elastic.co/t/how-to-create-a-new-beats-output/61074/8 "2016-09-26T11:00:34Z")

</div>

@ruflin, This is what I did to create a new output, build and test with filebeat

a) copied logstash output to my repo (ex: [github.com/ageetha/logmet](http://github.com/ageetha/logmet))  
b) changed package name and references from logstash to logmet ( I did  
this as my output is a modified version of logstash output)  
c) cloned this repo @ /root/src/github.com/  
d) cloned beats 1.3 @ /root/src/github.com  
e) modified /root/src/github.com/elastic/beats/filebeat/main.go to include imports to my repo  
f)  
modified /root/src/gitbub.com/elastic/beats/libbeat/outputs/outputs.go (  
inclded two new parameters I introduced in yml file )  
g) built filebeat

is there a way to introduce output specific yml parameters without  
having to modify libbeat/outputs/outputs.go. I would like to keep all my  
output specific code and config within my repo

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [September 29, 2016, 9:35am UTC](https://discuss.elastic.co/t/how-to-create-a-new-beats-output/61074/9 "2016-09-29T09:35:31Z")

</div>

There is no need to clone or copy filebeat. See my post from 7 days ago.

You might have 2 repositories for example:

1. `github.com/ageetha/logmet` holding the new output plugin
2. `github.com/ageetha/filebeat-logmet` holding your custom filebeat with logmet being included.

In `github.com/ageetha/filebeat-logmet` you have only one file named `main.go` with content:

```auto
import (
    "os"

    _ "github.com/ageetha/logmet"

    "github.com/elastic/beats/filebeat/beater"
    "github.com/elastic/beats/libbeat/beat"
)

Name := "filebeat-logmet"

func main() {
    if err := beat.Run(Name, "", beater.New()); err != nil {
        os.Exit(1)
    }
}

```

No run `go build` in `$GOPATH/src/github.com/ageetha/filebeat-logmet` directory and you will get your filebeat binary. Advantage of this solution is, you can more easily upgrade to new filebeat just by checking another branch from filebeat. Updating to filebeat 5.0 will require you to change main.go to:

```auto
import (
    "os"

        _ "github.com/myuser/mybeatsoutputs/myoutput"

    "github.com/elastic/beats/filebeat/beater"
    "github.com/elastic/beats/libbeat/beat"
)

func main() {
    if err := beat.Run("myfilebeat, "", beater.New); err != nil {
        os.Exit(1)
    }
}

```

Beats are made this way, to reduce the amount of maintenance normally required when copying and modifying some other projects source code.

---

<div class="post-metadata">

**Author:** ![Geetha\_Adinarayan](https://avatars.discourse-cdn.com/v4/letter/g/c6cbf5/32.png) [@Geetha\_Adinarayan](https://discuss.elastic.co/u/Geetha_Adinarayan)\
**Post date:** [October 6, 2016, 12:47pm UTC](https://discuss.elastic.co/t/how-to-create-a-new-beats-output/61074/10 "2016-10-06T12:47:33Z")

</div>

Thank you. This is helpful. I will try this.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 12, 2016, 4:07am UTC](https://discuss.elastic.co/t/how-to-create-a-new-beats-output/61074/11 "2016-10-12T04:07:10Z")

</div>

This topic was automatically closed after 21 days. New replies are no longer allowed.