Problem with logstash config

Elastic Stack Logstash
1

Hi all. We have trying to migrate our config (1.5) to newer (5.2) version of logstash . And we have some problem with multiline plugin. so here is an example of our config and log error.

filter {
if [type] == "osb_server" {
multiline {
pattern => "(^#)"
negate => true
what => "previous"
}
}
}

Error:

[2017-03-03T09:44:13,539][ERROR][logstash.agent ] Cannot load an invalid configuration {:reason=>"Couldn't find any filter plugin named 'multiline'. Are you sure this is correct? Trying to load the multiline filter plugin resulted in this error: Problems loading the requested plugin named multiline of type filter. Error: NameError NameError"}
[2017-03-03T09:44:16,533][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"filter", :name=>"multiline", :path=>"logstash/filters/multiline", :error_message=>"NameError", :error_class=>NameError, :error_backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:221:in namespace_lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:157:in legacy_lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:133:in lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:175:in lookup_pipeline_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:129:in lookup'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:95:in plugin'", "(eval):64:in initialize'", "org/jruby/RubyKernel.java:1079:in eval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:65:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:308:in reload_pipeline!'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:111:in reload_state!'", "org/jruby/RubyHash.java:1342:in each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:108:in reload_state!'", "org/jruby/ext/thread/Mutex.java:149:in synchronize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:107:in reload_state!'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:75:in execute'", "org/jruby/RubyProc.java:281:in call'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/interval.rb:20:in interval'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:75:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:271:in execute'", "org/jruby/RubyProc.java:281:in call'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/task.rb:24:in initialize'"]}
[2017-03-03T09:44:16,535][ERROR][logstash.agent ] Cannot load an invalid configuration {:reason=>"Couldn't find any filter plugin named 'multiline'. Are you sure this is correct? Trying to load the multiline filter plugin resulted in this error: Problems loading the requested plugin named multiline of type filter. Error: NameError NameError"}

Do we need to install filter plugin ? i think it goes from the box..

2

You need to change to the multiline codec instead.

3

thx for answer. we have changed and got this error.

filter {
if [type] == "osb_server" {
codec => multiline {
pattern => "(^#)"
negate => true
what => "previous"
}
}
}

Error:

[2017-03-03T09:50:28,505][ERROR][logstash.agent ] Cannot load an invalid configuration {:reason=>"Expected one of #, { at line 27, column 13 (byte 320) after filter {\nif [type] == "osb_server" {\n codec "}

In official manual use case is without codecs Multiline codec plugin | Logstash Reference [8.11] | Elastic

4

Codecs are specified within the input plugin, not as a separate filter. Where are you getting your data from?

5

quote from manual:

One more common example is C line continuations (backslash). Here’s how to do that:

filter {
multiline {
type => "somefiletype"
pattern => "\$"
what => "next"
}
}

in 1.5 multiline was filter plugin?
We are getting data from filebeats.
example of our 1.5 ver config:

input {
beats {
port => 5044
tags => "beats"
}
}
filter {

OSB_SERVER type

if [type] == "osb_server" {
multiline {
pattern => "(^#)"
negate => true
what => "previous"
}
}

OSB type

if [type] =~ "osb" {
if [message] =~ "http:SOAPAction"SendVeterinaryPermit"</http:SOAPAction>" {
grok {
match => { "message" => "(?<easu_permit_number>.+)" }
}
}
if [message] =~ "" {
grok {
match => { "message" => "(?<application_status>[^<]+)" }
}
}

OSB_MLT_* type

if [type] =~ "osb_mlt_*" {
multiline {
pattern => "^####<"
negate => true
what => "previous"
}
}

SOA_MLT_* type

if [type] =~ "soa_mlt_*" {
multiline {
pattern => "^####<"
negate => true
what => "previous"
}
}
and so on...
the output is to the gelf.

6

The multiline filter plugin has been deprecated as it required a single worker thread and had problems with multiple inputs getting mixed. It is always recommended to perform multiline processing as close to the source as possible, so if you are using Filebeat, I would recommend to move the multiline processing there instead of using a codec in Logstash.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.