Defaulting pipeline worker threads to 1

[Honda_fred_elk (Fredrick Yessaian)]

Hello,

I have following config for logstash..

input {
file {
path => "/appl/log/TestLogs/*.log"
start_position => "beginning"
sincedb_path => "/appl/log/sincedbloc/sincedbfile.txt"
}
}

filter{
multiline{
pattern => "[#|\d{4}"
negate => true
what => "previous"
}

grok{
match => {"message" => "%{DATESTAMP:time-stamp}%{SPACE}%{LOGLEVEL:log-level}%{SPACE}%{DATA:class}:%{SPACE}%{GREEDYDATA:log-message}" }
}
}

output {
elasticsearch { hosts => ["localhost:9200"] }
stdout{}
file{
path => "/appl/log/TestLogsOutput/LogStash_output.log"
}
}

As soon as I introduced multiline filter and restarted logstash.. I got following message..

Defaulting pipeline worker threads to 1 because there are some filters that might not work with multiple worker threads", :count_was=>4, :filters=>["multiline"], :level=>:warn}
blogs suggest to use FileBeat..

Can anyone advice is there any alternate or I should go with FileBeat?

Thanks
Fredrick

[Bevan (Bevan Bennett)]

You can use the multiline codec to the file input instead of having it as a separate filter.

As a standalone filter stage, multiline forces a single threaded approach because otherwise it can't tell which line came from which file.

We use:

file {
    type => "foobar"
    path => [ "/var/log/foobar/foo.log",
              "/var/log/foobar/bar.log",
              "/var/log/foobar/baz.log"
            ]
    codec => multiline {
        pattern => "^\d{4}-\d{2}-\d{2}\@\d{2}:\d{2}:\d{2}(\.\d{1,4})?"
        negate => true
        what => "previous"
    }
}

...for exactly that reason.