Logstash Pipeline grok filter (match) fails - Expected one of [ \\t\\r\\n], \"#\", \"{\", \"}\"

Elastic Stack Logstash
1

Hi Logstash community,

I'm trying to parse a log through logstash but there seems to be an issue with my grok pattern, however I have tried a lot of different things now and always get the exact same error message.

The Log I'm trying to parse

[
  {"name":"mmc.exe","pid":"6052","start_time":"1585318707"},
  {"name":"notepad++.exe","pid":"5888","start_time":"1585318573"},
  {"name":"firefox.exe","pid":"7456","start_time":"1585318484"},
  {"name":"firefox.exe","pid":"5560","start_time":"1585318360"},
  {"name":"firefox.exe","pid":"7668","start_time":"1585318348"}
]

My pipeline.conf

input {
  beats {
    port => "5044"
    ssl => true
    ssl_certificate_authorities => ["/usr/share/logstash/config/certs/ca.crt"]
    ssl_certificate => "/usr/share/logstash/config/certs/ls-server.crt"
    ssl_key => "/usr/share/logstash/config/certs/ls-server-pkcs8.key"
    ssl_verify_mode => "none"
  }
}

filter {
  if "demo" in [tags]{
  mutate {
    gsub => [ "message", "\[", "","message", "\]", "" ]
  }
  grok {
  match => { "message" => \{"name":%{QUOTEDSTRING:name},"pid":"%{NUMBER:pid}","start_time":"%{NUMBER:date}"\} }
  }
 }
}

Logstash wont boot and always throws this error:

Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", [A-Za-z0-9_-], '\"', \"'\", [A-Za-z_], \"-\", [0-9], \"[\", \"{\" at line 18, column 27 (byte 459) after filter {\n  if \"demo\" in [tags]{\n  mutate {\n    gsub => [ \"message\", \"\\[\", \"\",\"message\", \"\\]\", \"\" ]\n  }\n  grok {\n  match => { \"message\" => ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:58:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:66:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:28:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:27:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:181:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:67:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:43:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342:in `block in converge_state'"]}

The gsub part seems to be working fine, at least when I commented out the grok part the logstash booted without a problem.
Here is a link to the official documentation for the grok match plugin, maybe I'm reading it wrong but I cant seem to find any syntax error with my grok filter.
I debugged the GROK Pattern with the heroku debugger and it worked fine. I have also uploaded the log manually to Elasticsearchs Machine Learning tool and tried that grok filter out. Also to no avail.

2

You match a field against a string. The string has to be in quotes. Since you have double quotes in your pattern you should surround it in single quotes.

match => { "message" => '{"name":%{QUOTEDSTRING:name},"pid":"%{NUMBER:pid}","start_time":"%{NUMBER:date}"}' }
3

Hi Badger, I tried your solution but I'm still getting the same error message.

The only time I was able to successfully boot my logstash instance was when I got rid of the curly brackets so I am guessing Logstash/Grok might have some sort of issue with those.

4

Try this:
match => { "message" => '{"name":"%{DATA:name}","pid":"%{NUMBER:pid}","start_time":"%{NUMBER:date}"}' }

5

I was able to find a solution for the issue.
I tried to do multiple extractions within the syntax for a single extraction.

My new filter now looks like this:

filter {
  if "osquery" in [tags]{
  grok {
    match => {
      "message" => [
        '\"name\":%{QUOTEDSTRING:name}"',
        '\"pid\":%{NUMBER:pid}"',
        '\"start_time\":"%{NUMBER:date}"'
     ]

Badger and fadjar both were also correct in their usage of single ticks, with the new syntax and single ticks the logstash instance is now booting.