Custom filebeat to attach data on outgoing stream

[swapnil.bordiya]

I have a server with multiple log sources. To make better sense out of logs we need to put additional info by querying some data sources from a DB server and attach on the filebeats outgoing data stream.

Is there anyway to create a custom filebeats and intercept the outgoing data streams dynamically? For example in below log entry a new value dynamic field can be added:

<LogMessage time="2019-07-08 09:08:14.447" source="Equipment" logname="p2p" severity="FATAL" NewDynamicField="DynamicValue">
<![SAMPLE DATA]>

[steffens (Steffen Siering)]

Filebeat does not parse your logs, but processors can add additional metadata to events. The meta data are normally used via Logstash or Ingest Node to further modify the event.

Based on which information do you plan to enhance you event?

Does only filebeat has access to the DB server? If no, have you considered logstash? e.g. the jdbc_static or elasticsearch filter might fit your use case.

[swapnil.bordiya]

Hey Steffen, thanks for the response Yes only filebeat has access to the DB server which is blocking us to consider logstash.

We also have XML files containing the same metadata on the clients which filebeat can read but we run into same problem of not able to modify the events.

Can you explain more the processors? Do they allow to change events on filebeats? Also if there is absolutely no way through filebeats to achieve this functionality, is there a way to write a custom beats to modify events?

[steffens (Steffen Siering)]

Yes, you can create a customized filebeat by writing your own processor. Processors are executed before publishing events and can modify events. As your processor will be stateful, make sure it is thread safe + if you have some rich/nested object reused between many calls, use obj.Clone(), so to not give anyone the chance modify the object in subsequent processors.

See filebeat/main.go. One uses the 'import-trick' in order to build a customized filebeat:

package main

import (
	"os"

        _ "path/to/my/extension/plugin1"
        _ "path/to/my/extension/plugin2"
        ...

	"github.com/elastic/beats/filebeat/cmd"
)

func main() {
	if err := cmd.RootCmd.Execute(); err != nil {
		os.Exit(1)
	}
}

On linux and mac one can actually use go-plugins and load them dynamically. Here is a quite old example on how to do this: https://github.com/urso/beats-sample-plugins. Note: the example might not compile anymore, due to type changes in Beats, but the way one defines and loads a plugin hasn't changed at all. The essence is this line: https://github.com/urso/beats-sample-plugins/blob/master/plugins/all/main.go#L13

Problem with go-plugins is that both filebeat and the plugin needs to be compiled using the same compiler and very same checkout. Although you don't need to modify filebeat in this case, you are still required to build it yourself.

All beats processors are available in the libbeat/processors package: https://github.com/elastic/beats/tree/master/libbeat/processors

See add_fields processor source code: https://github.com/elastic/beats/blob/master/libbeat/processors/actions/add_fields.go

btw. if you want to reuse some code, the processors sample plugins do contain a lookup processor with caching support. The lookup.exec processor runs a script in order to collect additional meta data for events. This would allow you to write the database and xml lookup any language you like. See: https://github.com/urso/beats-sample-plugins/tree/master/lookup