Logstash 5.1.1 - Couldn't find any filter plugin named 'multiline'

[MikeSiz (Michael)]

Hi, I am trying to migrate from logstash 2.3.4 to newer 5.1.1.
After default installation (registering logstash 5.1 repo and yum install logstash)
I found that logstash cannot find "multiline' plugin.

I tried:
/usr/share/logstash/bin/logstash-plugin install logstash-codec-multiline

and it installed multiline successfully. Could see it in the list:

[root@xxxx logstash]# /usr/share/logstash/bin/logstash-plugin list | grep multil
logstash-codec-multiline

However, it still cannot find it, when I try to check configuration:

[root@xxxx logstash]# sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/logstash.conf --config.test_and_exit

it raises the error in /var/log/logstash/logstash-plain.log:

[root@xxxx logstash]# more logstash-plain.log

[2017-01-11T10:08:11,674][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"filter", :name=>"multiline", :path=>"logstash/filters/mult
iline", :error_message=>"NameError", :error_class=>NameError,
....

Could someone help me? What am I doing wrong?

Thanks,
Michael

[MikeSiz (Michael)]

I have found the problem.

I should have been installing "filter", not a "codec"

/usr/share/logstash/bin/logstash-plugin install logstash-filter-multiline

[Christian_Dahlqvist]

You should probably try to migrate away from the multiline filter as it has been deprecated. It is generally recommended to perform this processing as close to the sources as possible. Filebeat supports this and if you need to do it in Logstash, the multiline codec is the recommended way to go.

[MikeSiz (Michael)]

Thank you, will look at it

[seth.yes (Seth S)]

@Christian_Dahlqvist

It would be great if the multiline filter was marked as deprecated or just removed entirely from the current documentation.

edit: added screenshot

[Christian_Dahlqvist]

What you linked to is the multi line codec, not the multi line filter which has been deprecated. The multi line codec is still supported as it does not suffer from the limitations of the filter.

[seth.yes (Seth S)]

Understood, however the screenshot shows the multiline filter, which makes the multiline filter appear as if it's still supported.

[Christian_Dahlqvist]

Aha, did not notice that. We should get that corrected in that case.

[seth.yes (Seth S)]

@Christian_Dahlqvist I'm not in a good place to edit the docs but if you get anyone to change it, someone should also add that apparently the 'auto_flush_interval' is required to make the multiline codec work. Thanks!

Logstash input file codec multiline not working as expected

For me setting auto_flush_interval => 5 does solve the problem.

Issue with file input multiline codec

I found a related issue https://github.com/logstash-plugins/logstash-input-file/issues/90 which mentions that auto_flush needs to be set via the auto_flush_interval codec parameter. Once I set auto_flush_interval => 3 I observed the desired behavior. file { path => "c:/temp/test.log" start_position => beginning sincedb_path => "NUL" ignore_older => 0 codec => multiline { pattern => "^%{MONTHDAY} %{MONTH} %{YEAR} %{TIME}" negate => true what => previou…

[Christian_Dahlqvist]

I have opened an issue against the Github repository containing the Logstash documentation. I have also opened another issue against the multiline codec plugin to either set a default value or add additional information about the auto_flush_interval parameter and use to the documentation.

Thanks for pointing this out.

[rlv_praveen]

I also faced the same error in logstash 5.1.2 version. multiline filter is removed from logstash 5.. versions. you could please try the multiline codec filter instead.

You need to use the multiline codec in the input tag not in the filter tag. For sure your problem will be solved. Kudos try it and update us.

[seth.yes (Seth S)]

Thanks for your response. I've implemented the codec and am still not getting any data pulled in through Logstash. I'll start another thread in an attempt to not further-hijack this thread.