Require support for conditional statement within beats input section

Elastic Stack Beats
1

I have a use case wherein I am running 2 different prospectors for 2 different file types and shipping the log events to logstash. Now in my logstash while I receive events from both file types, I would however want for one particular event type to be using multiline codec else ignore for other event types. Currently there is no conditional support as such for this. See below for my expectation:

input {
beats {
port => "5033"
if[type] == "trc" {
codec => multiline {
negate => "true"
pattern => "(.MAJOR)"
what => "previous"
}
}
}
}

Is there a way I can achieve this? I also tried the following in my logstash config which seems very very legit but it just doesn't allow me to do it

input {
if[type] == "trc" {
beats {
port => "5044"
codec => multiline {
negate => "true"
pattern => "(.MAJOR)"
what => "previous"
}
}
}
else {
beats { port => "5044" }
}
}

For the above config I keep getting port already in use error. One possible way to get around this use case is to run another filebeat instance shipping those events alone on a different port. But I do not wish to do that. Could you please suggest if there is a way which I am unaware of to get around this use case? Else would this be considered as a requirement ?

2

Hello Vinod,

You will have to place this conditional in your filter section, and e.g. use a regex to identify the documents you want to drop and drop them with the drop filter.
Conditionals cannot be placed in the input section (and logically this will never be possible).

Another thought you might like to try would be to have two beats inputs each with a unique port, and in your filebeat agents send to the correct port knowing if the source is multiline or not.

Cheers,
-Robin-

3

Hi Robin,

Since my requirement is to use a multiline codec/filter, it wouldn't be possible for me to push the conditional statements into filter sections for two reasons:

  1. With multiline filter used, worker threads cannot be set to 2 or more
  2. I also use a uuid plugin. With multiline being used in filter section, it generates and returns me an array of uuid's instead of one. This is not acceptable for my application.

Guess as a last resort I need to run a dedicated filebeat instance shipping events to logstash indexer on a dedicated port for which I require multiline support.

4

@vinod8427 you can follow progress on multiline support on github.

Haven't used these to filter plugins myself yet, but can you apply the uuid plugin after multiline (maybe combined with conditional) to generate only one uuid or use the mutate filter to select only one uuid from this array?

5

Hi Vinod,
Was looking for an answer to that myself and wound up using filebeats multi-line support to agggregate the lines before shipping ( a stack trace in my case) and then do the groking and mutating based of a conditional filter in logstash itself. I know this is an old item, but maybe it will help someone going through the same frustration.
Marc

2 Likes