Logstash filter multiline not working

tharu85 · July 5, 2017, 11:22am

I have a java log which has multiline in the log entries. So I have used multiline in filter function in logstash as below.

input {
     file {
            path => "/etc/logstash-5.2.2/sample7.log"
	start_position => beginning		
     }
}

filter {
multiline {
	negate => true
	pattern => "^%{TIMESTAMP_ISO8601}"		
	what => "previous"
}

grok {
	match => ["message", "\[%{TIMESTAMP_ISO8601:java_timestamp}\] %{LOGLEVEL:severity} +%{JAVACLASS:java_class} : %{GREEDYDATA:message}",
              "message", "%{JAVASTACKTRACEPART}",
              "message", "%{JAVACLASS:java_class}Exception(: %{GREEDYDATA:logmessage})?",
			  "message", "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log_level} +\[%{DATA:logger}\] %{GREEDYDATA:massage_data}"
             ]
   }	
 }


 output {
      stdout { codec => rubydebug }
 }

But above code gives following error

Sending Logstash's logs to /etc/logstash-5.2.2/logs which is now configured via log4j2.properties
[2017-07-05T16:05:43,638][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"filter", :name=>"multiline", :path=>"logstash/filters/multiline", :error_message=>"NameError", :error_class=>NameError, :error_backtrace=>["/etc/logstash-5.2.2/logstash-core/lib/logstash/plugins/registry.rb:221:in namespace_lookup'", "/etc/logstash-5.2.2/logstash-core/lib/logstash/plugins/registry.rb:157:inlegacy_lookup'", "/etc/logstash-5.2.2/logstash-core/lib/logstash/plugins/registry.rb:133:in lookup'", "/etc/logstash-5.2.2/logstash-core/lib/logstash/plugins/registry.rb:175:inlookup_pipeline_plugin'", "/etc/logstash-5.2.2/logstash-core/lib/logstash/plugin.rb:129:in lookup'", "/etc/logstash-5.2.2/logstash-core/lib/logstash/pipeline.rb:95:inplugin'", "(eval):12:in initialize'", "org/jruby/RubyKernel.java:1079:ineval'", "/etc/logstash-5.2.2/logstash-core/lib/logstash/pipeline.rb:65:in initialize'", "/etc/logstash-5.2.2/logstash-core/lib/logstash/pipeline.rb:144:ininitialize'", "/etc/logstash-5.2.2/logstash-core/lib/logstash/agent.rb:275:in create_pipeline'", "/etc/logstash-5.2.2/logstash-core/lib/logstash/agent.rb:95:inregister_pipeline'", "/etc/logstash-5.2.2/logstash-core/lib/logstash/runner.rb:264:in execute'", "/etc/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:inrun'", "/etc/logstash-5.2.2/logstash-core/lib/logstash/runner.rb:183:in run'", "/etc/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:inrun'", "/etc/logstash-5.2.2/lib/bootstrap/environment.rb:71:in `(root)'"]}
[2017-07-05T16:05:43,662][ERROR][logstash.agent ] Cannot load an invalid configuration {:reason=>"Couldn't find any filter plugin named 'multiline'. Are you sure this is correct? Trying to load the multiline filter plugin resulted in this error: Problems loading the requested plugin named multiline of type filter. Error: NameError NameError"}

How do I place multiline inside filter function properly, but not in input function ?

magnusbaeck · July 5, 2017, 11:30am

You don't have the multiline filter plugin installed. That plugin is deprecated. Use the multiline codec instead.

https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html

tharu85 · July 5, 2017, 11:58am

@magnusbaeck,

Thanks, after installed the plugin, filter worked fine .

And I need some advice on removing following beats code from my log

 "@timestamp" => 2017-07-05T11:46:14.319Z,
    "@version" => "1",
        "beat" => {
    "hostname" => "graylog-server-one",
        "name" => "graylog-server-one",
     "version" => "5.4.0"
  },
        "host" => "graylog-server-one",
   "timestamp" => "2017-05-12T14:17:23.479+05:30"

I need to remove following block from the log. I tried with if condition, But it does't remove that part

 "beat" => {
    "hostname" => "graylog-server-one",
        "name" => "graylog-server-one",
     "version" => "5.4.0"
  }

magnusbaeck · July 5, 2017, 12:20pm

Have you tried using a mutate filter and its remove_field option?

I tried with if condition, But it does't remove that part

Please always show exactly what you attempted.

tharu85 · July 6, 2017, 4:02am

@magnusbaeck

remove_field

does not removed

"beat" => {
    "hostname" => "graylog-server-one",
    "name" => "graylog-server-one",
    "version" => "5.4.0"
}

block.

I have tried following syntax but it don't remove any fields

mutate 
   {
		remove_field  => { "[beat][version]" }
   }


mutate 
   {
	    remove_field =>  [ "beat", "version" ]
   }

magnusbaeck · July 6, 2017, 8:52am

Do you try this?

remove_field  => ["[beat][version]"]

tharu85 · July 6, 2017, 9:39am

I tried given code. But it doesn't remove the mentioned fields

mutate {
	remove_field  => ["[beat][version]","[beat][hostname]" ]
}

Below is the beat log part that I need to remove from my parse log

 "beat" => {
        "hostname" => "graylog-server-one",
        "name" => "graylog-server-one",
        "version" => "5.4.0"
  },

magnusbaeck · July 6, 2017, 8:04pm

Works fine for me:

$ cat data 
{"beat": {"hostname": "graylog-server-one", "name": "graylog-server-one", "version": "5.4.0"}}
$ cat test.config 
input { stdin { codec => json } }
output { stdout { codec => rubydebug } }
filter {
  mutate {
    remove_field => ["[beat][version]"]
  }
}
$ logstash -f test.config < data 
Settings: Default pipeline workers: 8
Pipeline main started
{
          "beat" => {
        "hostname" => "graylog-server-one",
            "name" => "graylog-server-one"
    },
      "@version" => "1",
    "@timestamp" => "2017-07-06T20:03:48.188Z",
          "host" => "bertie"
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}

tharu85 · July 7, 2017, 3:28am

Here is my code. But below code doesn't remove the version filed under beat

input {
   beats {
       port => 5052
   }
}

filter {
multiline {
	negate => true
	pattern => "^%{TIMESTAMP_ISO8601}"		
	what => "previous"
}

if "beats_input_codec_plain_applied" in [tags] {
	mutate {
		remove_tag => ["beats_input_codec_plain_applied"]
	}
}

grok {
	match => ["message", "\[%{TIMESTAMP_ISO8601:java_timestamp}\] %{LOGLEVEL:severity} +%{JAVACLASS:java_class} : %{GREEDYDATA:message}",
              "message", "%{JAVASTACKTRACEPART}",
              "message", "%{JAVACLASS:java_class}Exception(: %{GREEDYDATA:logmessage})?",
			  "message", "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log_level} +\[%{DATA:logger}\] %{GREEDYDATA:massage_data}"
             ]
}	

mutate {
	remove_field  => [ "[beat][version]" ]
  }
}

output {
    stdout { codec => rubydebug }
}

Below is the sample output, see -> "version" => "5.4.0" is there in the output

],
"@timestamp" => 2017-07-07T03:23:04.738Z,
      "file" => "NioClientBoss.java",
  "@version" => "1",
      "beat" => {
    "hostname" => "graylog-server-one",
        "name" => "graylog-server-one",
     "version" => "5.4.0"
},
      "host" => "graylog-server-one",
     "class" => "org.jboss.netty.channel.socket.nio.NioClientBoss"
}

magnusbaeck · July 11, 2017, 6:46pm

Well, my configuration works and yours doesn't. I suggest you transform your configuration to my configuration step by step until the [beat][version] field disappears. Start with the multiline filter.

And, I repeat, do not use the multiline filter. At all. Use Filebeat's multiline feature instead.

system · August 8, 2017, 6:47pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.