Hi all,
I'm interested in using filebeat as a library in another application. I would like to be able to intercept its log messages and include them in the parent application. Is there any recommended way to do this?
Thanks,
Thomas
Uhm, is you application written in golang? You mean library in a sense of importing the filebeat package? Or do you plan to run filebeat as child-process and capture stdout (events being published) and stderr (filebeat logs)?
Hi Steffen. Yes, written in golang and being used as a library in the sense that various packages from the filebeat project will be imported.
Might be somewhat tricky, but possible. Depending on the amount of integration you're seeking.
All beats have kind of a constructor of type (see libbeat/beat.go):
// Creator initializes and configures a new Beater instance used to execute
// the beat its run-loop.
type Creator func(*Beat, *common.Config) (Beater, error)
In theory multiple instances can be generated. The Beat interface implements only Run and Stop (used to kill filebeat on signal).
Constructing the configuration and the Beat type from code might be a little tricky. One could use a template in go and pass it to the config parser.
Unfortunately the logging in beats is still 'global', that is you have to configure it as well.
The Publisher and 'client' in the Beat object is fortunately an interface. That is, you can capture filebeat events and bypass the libbeat publisher pipeline (doesn't need to configured).
To instantiate a filebeat instance with correct settings you basically have to redo some of the heavy-lifting done by beat.Run.
If you basically want to have filebeat with custom output, you can just import filebeat as package plus your custom output. Here is an example how this can be done: https://github.com/raboof/beats-output-http
If all you want is custom processing, you can do the same import trick as in the output as well (internally outputs/processor are plugins to beats).
If you're fine with your application being a beat itself, you can 'layer' filebeat. You can introduce your own beat constructor creating an intermediate Beat object with custom Publisher. Then all the setup is still handled by libbeat, but you can intercept filebeat events in your own publisher type.
Central to all configuration management in beats is common.Config, which is based on go-ucfg. It works somewhat like json.Marshal/Unmarshal, but uses it's own internal representation and adds some more features. Using Merge and Unpack, you can easily build your configurations in code using map[string]interface{} (or any other valid type).
In case you're looking for adding custom prospector types/parsers, filebeat code-base is not there yet. We have ideas to refactor filebeat, so readers are pluggable from within go, as is currently possible with outputs and processors.
Wow, thanks for the detailed information. This has been helpful to confirm my understanding. To simplify things, I have essentially just used the Crawler and the Registrar to create my own custom loop with custom spooler that does it's own publishing (I had seen beats-output-http, but as this program already has a lot of http posting infrastructure, it would have been messy to integrate). In all, you're right that this required re-doing some amount of the heavy lifting already implemented by filebeat.
In terms of the logging, I've decided to keep it simple and just let filebeat log to its own file for now. Thanks for the help.
Sounds legit.
Note: Crawler and Registrar are very private functionality and prone to change.Reason I proposed using filebeat.New, it's the most interfacing required between filebeat and your project. We're planing some 'heavy' refactoring including the publisher pipeline and how prospectors are done (more pluggable and reusable), also moving the spooler away from filebeat into the libbeat publisher pipeline. Advantage of these changes is, integration might become easier. Disadvantage is, you will have to do it again. But given you have/want your own spooler, filebeat.New might indeed not the right interface right now. After refactoring filebeat.New you will either use filebeat.New or use/import the prospectors only (the later sounds more convenient).
Well, there is currently no timeline for these changes/refactoring and quite a big change. But it's something you might have to deal with at some point in the future. For this reason would vendor one of the 5.x branches, to get a somewhat stable API for now.
Yup, I'm vendoring 5.3.1 currently. But I will keep a lookout for these
changes as well, thanks so much for the heads up!
This topic was automatically closed after 21 days. New replies are no longer allowed.