Hi Team,
I'm trying to use the grok filter in the logstash configuration pipeline. The grok pattern seem to work in the kibana provided grok debugger. But using the pattern in the filter throws the error that logstash configuration error.
Logstash Config filter
filter {
if [path] == "/media/sf_ELK/ATG01log.txt" {
grok {
match => { "message" => "(?%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}) (?(?:[A-Z]+)) (?(?:[[A-Za-z.]+])) (?(?:[a-zA-Z0-9]+)) (?(?:[a-zA-Z0-9]{7}-[a-zA-Z0-9]{18}-[a-zA-Z0-9]{3}-[a-zA-Z0-9]{9}-[a-zA-Z0-9]{11}!-[0-9]{9}![0-9]{13})) (?(?:[a-z0-9]+)) (?(?:AEO_(CA|INTL|US))) (?(?:(en|fr)_(US|CA))) (?<request_type>(?:[A-Z]+)) (?<request_path>(?:[/a-zA-Z0-9]+)) (?<active_thread>(?:([.'))) (?<thread_info>(?:.],)) %{GREEDYDATA:body}"}
tag_on_failure => ["grok-parse-failed"]
}
kv {
trimkey => "\s"
field_split => ",?"
}
}
}
Sample Log
2019-07-24 08:29:54,838 INFO [nucleusNamespace.aeo.service.jaxrs.LoggingResponseFilter] 80f4dd01fbed4e688ab24e822866c08a 87Yj9Zv-GlrPJkesCNodmLHg3S-VgY-rbdwfebRc-C2cSDL0JSkX!-211948433!1563971394558 ugp417017865 XXX_CA en_US POST /public/v1/currentUser/Login ([ACTIVE] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)') {thread=Thread[[ACTIVE] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)',5,Pooled Threads], threadId=91, sessionId=null, profileId=ugp417017816, userEmail=test@test.com, customerIPAddress=127.0.0.1, User-Agent=Mozilla/6.3.0.1365-Develop (iPhone; iOS 11.2.5; Scale/3.0), x-forwarded-for=127.0.0.1} Authentication status=200, loginID=test@test.com CAP Route End: route=/public/v1/currentUser/Login#POST, totalTime=273 ms
Logstash Exception
Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 11, column 542 (byte 734) after filter { \n if [path] == "/media/sf_ELK/ATG01log.txt" {\n\tgrok {\n\t\tmatch => { "message" => {'(?%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}) (?(?:[A-Z]+)) (?(?:\[[A-Za-z\.]+\])) (?(?:[a-zA-Z0-9]+)) (?(?:[a-zA-Z0-9]{7}\-[a-zA-Z0-9]{18}\-[a-zA-Z0-9]{3}\-[a-zA-Z0-9]{9}\-[a-zA-Z0-9]{11}\!\-[0-9]{9}\![0-9]{13})) (?(?:[a-z0-9]+)) (?(?:AEO_(CA|INTL|US))) (?(?:(en|fr)_(US|CA))) (?<request_type>(?:[A-Z]+)) (?<request_path>(?:[\/a-zA-Z0-9]+)) (?<active_thread>(?:\(\[.\'\))) (?<thread_info>(?:.\]\,)) %{GREEDYDATA:body}'", :backtrace=>["/media/sf_ELK/logstash-7.0.1/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative'", "/media/sf_ELK/logstash-7.0.1/logstash-core/lib/logstash/compiler.rb:49:incompile_graph'", "/media/sf_ELK/logstash-7.0.1/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2577:inmap'", "/media/sf_ELK/logstash-7.0.1/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:ininitialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in initialize'", "/media/sf_ELK/logstash-7.0.1/logstash-core/lib/logstash/java_pipeline.rb:23:ininitialize'", "/media/sf_ELK/logstash-7.0.1/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "/media/sf_ELK/logstash-7.0.1/logstash-core/lib/logstash/agent.rb:325:inblock in converge_state'"]}