Grok Pattern Help Please

[jenyphur (Jennifer Campion)]

Hi there, pretty new to Logstash and Grok patterns. I am trying to parse out a custom log file and here is what I'm trying to do.

Log Sample;
<15>1 2018-08-09T07:37:48.306-05:00 MacBook-Pro.local ReaccomTask - Audit [mdc@18060 app_className="TDSSabreConnection" app_client_tranid="D228E0B6192E4C748CCCDDD5494B4698" app_loglevel="DEBUG" app_recordLoc="UANNFR" app_servername="QueueMoveWorker" app_timestamp="2018-08-09T07:37:48,306" app_tranid="BD15A16391FC4C3EB1BA840D6BE2C03F" app_version="Thread-8" cf_offset="706"] <>

I am having trouble taking everything after ReaccomTask to the beginning of app_className and essentially ignoring it. Then I am trying to get each field after that parsed out using the field name provided.

Using an online grok tester I was able to get the result I'm looking for

Using this ugly grok pattern:

%{TIMESTAMP_ISO8601:timestamp} %{DATA:host} %{DATA:app_name} %{DATA:app_notsure} app_className="%{NOTSPACE:app_className}&quot; app_client_tranid="%{NOTSPACE:app_client_tranid}&quot; app_loglevel="%{NOTSPACE:app_loglevel}&quot; app_recordLoc="%{NOTSPACE:app_recordLoc}&quot; app_servername="%{NOTSPACE:app_servername}&quot; app_timestamp="%{NOTSPACE:app_timestamp}&quot; app_tranid="%{NOTSPACE:app_tranid}&quot; app_version="%{NOTSPACE:app_version}&quot; cf_offset="%{NOTSPACE:cf_offset}&quot;]%{GREEDYDATA:app_message}

But when I try to use this in logstash , I keep getting these errors:
[2018-08-10T08:54:47,395][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 17, column 128 (byte 291) after filter\n{\n\n if [type] == "syslog"\n {\n\n grok\n {\n\t\tmatch => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{DATA:host} %{DATA:app_name} %{DATA:app_notsure} app_className="", :backtrace=>["C:/Logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "C:/Logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "C:/Logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "C:/Logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "C:/Logstash/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "C:/Logstash/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "C:/Logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "C:/Logstash/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "C:/Logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "C:/Logstash/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "C:/Logstash/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "C:/Logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "C:/Logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "C:/Logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "C:/Logstash/logstash-core/lib/logstash/agent.rb:105:in block in execute'", "C:/Logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:18:ininterval'", "C:/Logstash/logstash-core/lib/logstash/agent.rb:94:in execute'", "C:/Logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "C:/Logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

So something is clearly wrong.

Any help would be GREATLY appreciated.

Jennifer

[Badger]

You need to escape the double quote immediately after app_className.

Personally I would use dissect for this, not grok.

[jenyphur (Jennifer Campion)]

Excellent..I actually removed all the double quotes and it seems to be working now

I'm not sure with dissect is, but I'll definitely look into it.

thanks!

[Badger]

Looking just at the part inside the square brackets. I would grok that to extract the key/value pairs then use a kv filter.

    grok { match => { "message" => "\[(?<mailperhaps>[^ ]+) %{DATA:restOfLine}\]" } } 
    kv { source => "restOfLine" }