Logstash filter for multiple log files parsing

[sam281 (sam)]

Hi,

I have file beats sending data to logstash. I am trying to filter and parse the log records based on the source file.

On kibana I see a field called "source" which has the source log file name for each record, I would like to leverage this field and write the parsing in logstash.

This is what i tried in logstash.conf which seems it is not working and logstash failed to start. Any pointers are highly appreciated. Thank you for your help!

filter{

if [source] = "/opt/IBM/log/SPI.log" {
	grok {
		match => { "message" => "%{TIMESTAMP_ISO8601:ibmtime}: %{LOGLEVEL:severity}: %{DATA:messageid}: %{GREEDYDATA:message}"}
		overwrite => [ "message" ]
	}
}

}

[sam281 (sam)]

Here is the command output that I see when I try to run the logstash manually.

C:\ELK-Stack\logstash\bin>logstash -f ..\logstash.conf
io/console not supported; tty will not be manipulated
Settings: Default pipeline workers: 4
←[31mPipeline aborted due to error {:exception=>#<Errno::EADDRINUSE: Address alr
eady in use - bind - Address already in use: bind>, :backtrace=>["org/jruby/ext/
socket/RubyTCPServer.java:118:in initialize'", "org/jruby/RubyIO.java:853:inn
ew'", "C:/ELK-Stack/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2
.2.8/lib/lumberjack/beats/server.rb:51:in initialize'", "C:/ELK-Stack/logstash/ vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.2.8/lib/logstash/inputs/beat s.rb:119:inregister'", "C:/ELK-Stack/logstash/vendor/bundle/jruby/1.9/gems/log
stash-core-2.3.2-java/lib/logstash/pipeline.rb:330:in start_inputs'", "org/jrub y/RubyArray.java:1613:ineach'", "C:/ELK-Stack/logstash/vendor/bundle/jruby/1.9
/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:329:in start_inputs'", "C:/ELK-Stack/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib /logstash/pipeline.rb:180:instart_workers'", "C:/ELK-Stack/logstash/vendor/bun
dle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:136:in run '", "C:/ELK-Stack/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java /lib/logstash/agent.rb:465:instart_pipeline'"], :level=>:error}←[0m
stopping pipeline {:id=>"main"}
←[33mAn unexpected error occurred! {:error=>#<NoMethodError: undefined method e ach' for nil:NilClass>, :class=>"NoMethodError", :backtrace=>["C:/ELK-Stack/logs tash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.2.8/lib/logstash/inputs /beats.rb:188:instop'", "C:/ELK-Stack/logstash/vendor/bundle/jruby/1.9/gems/lo
gstash-core-2.3.2-java/lib/logstash/inputs/base.rb:83:in do_stop'", "org/jruby/ RubyArray.java:1613:ineach'", "C:/ELK-Stack/logstash/vendor/bundle/jruby/1.9/g
ems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:385:in shutdown'", "C:/EL K-Stack/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logst ash/agent.rb:411:instop_pipeline'", "C:/ELK-Stack/logstash/vendor/bundle/jruby
/1.9/gems/logstash-core-2.3.2-java/lib/logstash/agent.rb:404:in shutdown_pipeli nes'", "org/jruby/RubyHash.java:1342:ineach'", "C:/ELK-Stack/logstash/vendor/b
undle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/agent.rb:404:in shut down_pipelines'", "C:/ELK-Stack/logstash/vendor/bundle/jruby/1.9/gems/logstash-c ore-2.3.2-java/lib/logstash/agent.rb:400:inshutdown'", "C:/ELK-Stack/logstash/
vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/agent.rb:229:
in execute'", "C:/ELK-Stack/logstash/vendor/bundle/jruby/1.9/gems/logstash-core -2.3.2-java/lib/logstash/runner.rb:94:inrun'", "org/jruby/RubyProc.java:281:in
call'", "C:/ELK-Stack/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3. 2-java/lib/logstash/runner.rb:99:inrun'", "org/jruby/RubyProc.java:281:in cal l'", "C:/ELK-Stack/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/ta sk.rb:24:ininitialize'"], :level=>:warn}←[0m
The signal HUP is in use by the JVM and will not work correctly on this platform

C:\ELK-Stack\logstash\bin>

[warkolm (Mark Walkom)]

@sam281

bind - Address already in use: bind

Showing your entire config would be useful, it might highlight what is causing this.