# Intercepting filebeat logs **URL:** \ **Category:** Beats\ **Created:** [April 25, 2017, 5:22pm UTC](https://discuss.elastic.co/t/intercepting-filebeat-logs/83604 "2017-04-25T17:22:36Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![tdelacour](https://avatars.discourse-cdn.com/v4/letter/t/f08c70/32.png) [@tdelacour](https://discuss.elastic.co/u/tdelacour)\ **Post date:** [April 25, 2017, 5:22pm UTC](https://discuss.elastic.co/t/intercepting-filebeat-logs/83604/1 "2017-04-25T17:22:36Z")
Hi all, I'm interested in using filebeat as a library in another application. I would like to be able to intercept its log messages and include them in the parent application. Is there any recommended way to do this? Thanks, Thomas ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [April 26, 2017, 9:57am UTC](https://discuss.elastic.co/t/intercepting-filebeat-logs/83604/2 "2017-04-26T09:57:21Z")
Uhm, is you application written in golang? You mean library in a sense of importing the filebeat package? Or do you plan to run filebeat as child-process and capture stdout (events being published) and stderr (filebeat logs)? ---
**Author:** ![tdelacour](https://avatars.discourse-cdn.com/v4/letter/t/f08c70/32.png) [@tdelacour](https://discuss.elastic.co/u/tdelacour)\ **Post date:** [April 26, 2017, 2:00pm UTC](https://discuss.elastic.co/t/intercepting-filebeat-logs/83604/3 "2017-04-26T14:00:44Z")
Hi Steffen. Yes, written in golang and being used as a library in the sense that various packages from the filebeat project will be imported. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [April 26, 2017, 2:52pm UTC](https://discuss.elastic.co/t/intercepting-filebeat-logs/83604/4 "2017-04-26T14:52:25Z")
Might be somewhat tricky, but possible. Depending on the amount of integration you're seeking. All beats have kind of a constructor of type (see libbeat/beat.go): ```auto // Creator initializes and configures a new Beater instance used to execute // the beat its run-loop. type Creator func(*Beat, *common.Config) (Beater, error) ``` In theory multiple instances can be generated. The Beat interface implements only `Run` and `Stop` (used to kill filebeat on signal). Constructing the configuration and the `Beat` type from code might be a little tricky. One could use a template in go and pass it to the config parser. Unfortunately the logging in beats is still 'global', that is you have to configure it as well. The Publisher and 'client' in the `Beat` object is fortunately an interface. That is, you can capture filebeat events and bypass the libbeat publisher pipeline (doesn't need to configured). To instantiate a filebeat instance with correct settings you basically have to redo some of the heavy-lifting done by [beat.Run](https://github.com/elastic/beats/blob/master/libbeat/beat/beat.go#L151). If you basically want to have filebeat with custom output, you can just import filebeat as package plus your custom output. Here is an example how this can be done: [https://github.com/raboof/beats-output-http](https://github.com/raboof/beats-output-http) If all you want is custom processing, you can do the same import trick as in the output as well (internally outputs/processor are plugins to beats). If you're fine with your application being a beat itself, you can 'layer' filebeat. You can introduce your own beat constructor creating an intermediate `Beat` object with custom `Publisher`. Then all the setup is still handled by libbeat, but you can intercept filebeat events in your own publisher type. Central to all configuration management in beats is [common.Config](https://github.com/elastic/beats/blob/master/libbeat/common/config.go#L34), which is based on [go-ucfg](https://github.com/elastic/go-ucfg). It works somewhat like json.Marshal/Unmarshal, but uses it's own internal representation and adds some more features. Using [Merge](https://godoc.org/github.com/elastic/go-ucfg#Config.Merge) and [Unpack](https://godoc.org/github.com/elastic/go-ucfg#Config.Unpack), you can easily build your configurations in code using `map[string]interface{}` (or any other valid type). ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [April 26, 2017, 2:55pm UTC](https://discuss.elastic.co/t/intercepting-filebeat-logs/83604/5 "2017-04-26T14:55:41Z")
In case you're looking for adding custom prospector types/parsers, filebeat code-base is not there yet. We have ideas to refactor filebeat, so readers are pluggable from within go, as is currently possible with outputs and processors. ---
**Author:** ![tdelacour](https://avatars.discourse-cdn.com/v4/letter/t/f08c70/32.png) [@tdelacour](https://discuss.elastic.co/u/tdelacour)\ **Post date:** [April 27, 2017, 1:38pm UTC](https://discuss.elastic.co/t/intercepting-filebeat-logs/83604/6 "2017-04-27T13:38:02Z")
Wow, thanks for the detailed information. This has been helpful to confirm my understanding. To simplify things, I have essentially just used the `Crawler` and the `Registrar` to create my own custom loop with custom spooler that does it's own publishing (I had seen beats-output-http, but as this program already has a lot of http posting infrastructure, it would have been messy to integrate). In all, you're right that this required re-doing some amount of the heavy lifting already implemented by filebeat. In terms of the logging, I've decided to keep it simple and just let filebeat log to its own file for now. Thanks for the help. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [April 28, 2017, 9:26am UTC](https://discuss.elastic.co/t/intercepting-filebeat-logs/83604/7 "2017-04-28T09:26:59Z")
Sounds legit. Note: `Crawler` and `Registrar` are very private functionality and prone to change.Reason I proposed using `filebeat.New`, it's the most interfacing required between filebeat and your project. We're planing some 'heavy' refactoring including the publisher pipeline and how prospectors are done (more pluggable and reusable), also moving the spooler away from filebeat into the libbeat publisher pipeline. Advantage of these changes is, integration might become easier. Disadvantage is, you will have to do it again. But given you have/want your own spooler, `filebeat.New` might indeed not the right interface right now. After refactoring `filebeat.New` you will either use `filebeat.New` or use/import the prospectors only (the later sounds more convenient). Well, there is currently no timeline for these changes/refactoring and quite a big change. But it's something you might have to deal with at some point in the future. For this reason would vendor one of the 5.x branches, to get a somewhat stable API for now. ---
**Author:** ![Thomas\_Delacour](https://avatars.discourse-cdn.com/v4/letter/t/8baadc/32.png) [@Thomas\_Delacour](https://discuss.elastic.co/u/Thomas_Delacour)\ **Post date:** [April 28, 2017, 12:18pm UTC](https://discuss.elastic.co/t/intercepting-filebeat-logs/83604/8 "2017-04-28T12:18:26Z")
Yup, I'm vendoring 5.3.1 currently. But I will keep a lookout for these changes as well, thanks so much for the heads up! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 16, 2017, 5:29pm UTC](https://discuss.elastic.co/t/intercepting-filebeat-logs/83604/9 "2017-05-16T17:29:44Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.