Tail once - route output based on filter

[Tomer_Yakir]

Hi,

I have the following use case:
Using Filebeat to tail a single log file. Based on the type of log entry detected I need to route the data to a different custom processor.
For example:

• Lines that start with A would be stored as is
• Lines that start with B would be pre-aggregated (count the number of lines of this type), then stored somewhere else

Note that I'm using Filebeat as a "go" library.
Currently what I'm doing is define a single prospector with multiple entries on "include_lines", then evaluate conditions on the Outlet to determine how to route the event.
Ideally I would be able to evaluate once instead of twice. Is that possible?

[steffens (Steffen Siering)]

You want to route events to different outputs (logstash, elasticsearch) or do you need some way of filtering/processing only? The later can be done in configuration only. Processors support conditions (regex, string match, ...).

[Tomer_Yakir]

Hi Steffen,

So, I believe it's a combination of both.
For a given log entry, I'd like to do something like (pseudo-code):

if line.contains("A") {
     dict = custom_aggregate(line, dict)
} else if line.contains("B") {
    store_in_custom_system_a(line)
} else if line.contains("C") {
     store_in_custom_system_b(line)
}

[steffens (Steffen Siering)]

Some filtering can be done in filebeat using processors with conditionals (See docs with examples and list of available processors).

Filebeat does not support event routing. The closes to event routing is a configurable index name for the Elasticsearch output (or kafka topic). If you need more routing capabilities based on actual contents, you would have to use Logstash.