Multiline issue

Elastic Stack Logstash
1

Hi everyone. In Logstash 1.5 I had this config working:

input {
lumberjack {
port => 5000
ssl_certificate => "/certs/logstash-forwarder.crt"
ssl_key => "/certs/logstash-forwarder.key"
}
}

filter {
if [type] == "type1" or [type] == "type2" {
multiline {
pattern => "^\d{4}:\d{2}:\d{2}"
negate => true
what => "previous"
}
} else if [type] == "type3" or [type] == "type4" {
multiline {
pattern => "^####"
negate => true
what => "previous"
}
}
}

output {
elasticsearch { hosts => "localhost:8200" }
stdout { codec => rubydebug }

but now in Logstash 2.0 it complains about multiline in filter section, so I guess I have to switch to codec multiline in input section, but there are no IFs.

How can I manage multilining of different types of logs?

P/S/ types are set in logstash-forwarder on multiple hosts

2

@11191 please use -w 1 flag when you start Logstash to force the # of workers to 1. By default in 2.0, we set the number of workers to half of the # of cores, but since you are using multiline filter which is not thread safe, you'll have to go back to 1 filter workers.

I'll update our documentation

1 Like
3

I think I'm seeing similar issue with the aggregate filter. It's using a mutex but the threads still seem to be stepping on each other. I'm working on an unrelated change to that plugin and could maybe address this as well but I'm not much of a rubyist. Is there a recommended way to handle multi-line/multi-event processing in filter plugins?

4

I confirm that "-w 1" option is recommended for aggregate filter.

Even if aggregate code is technically protected against multithreaded access, multiple events at the same time mean you don't have control on events order.
And as aggregate filter is dependent on events order to do its job, it requires only one worker to be sure that it behaves nicely.