Logstash Pipeline grok filter (match) fails - Expected one of [ \\t\\r\\n], \"#\", \"{\", \"}\"

Hi Logstash community,

I'm trying to parse a log through logstash but there seems to be an issue with my grok pattern, however I have tried a lot of different things now and always get the exact same error message.

The Log I'm trying to parse

[
  {"name":"mmc.exe","pid":"6052","start_time":"1585318707"},
  {"name":"notepad++.exe","pid":"5888","start_time":"1585318573"},
  {"name":"firefox.exe","pid":"7456","start_time":"1585318484"},
  {"name":"firefox.exe","pid":"5560","start_time":"1585318360"},
  {"name":"firefox.exe","pid":"7668","start_time":"1585318348"}
]

My pipeline.conf

input {
  beats {
    port => "5044"
    ssl => true
    ssl_certificate_authorities => ["/usr/share/logstash/config/certs/ca.crt"]
    ssl_certificate => "/usr/share/logstash/config/certs/ls-server.crt"
    ssl_key => "/usr/share/logstash/config/certs/ls-server-pkcs8.key"
    ssl_verify_mode => "none"
  }
}

filter {
  if "demo" in [tags]{
  mutate {
    gsub => [ "message", "\[", "","message", "\]", "" ]
  }
  grok {
  match => { "message" => \{"name":%{QUOTEDSTRING:name},"pid":"%{NUMBER:pid}","start_time":"%{NUMBER:date}"\} }
  }
 }
}

Logstash wont boot and always throws this error:

Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", [A-Za-z0-9_-], '\"', \"'\", [A-Za-z_], \"-\", [0-9], \"[\", \"{\" at line 18, column 27 (byte 459) after filter {\n  if \"demo\" in [tags]{\n  mutate {\n    gsub => [ \"message\", \"\\[\", \"\",\"message\", \"\\]\", \"\" ]\n  }\n  grok {\n  match => { \"message\" => ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:58:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:66:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:28:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:27:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:181:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:67:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:43:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342:in `block in converge_state'"]}

The gsub part seems to be working fine, at least when I commented out the grok part the logstash booted without a problem.
Here is a link to the official documentation for the grok match plugin, maybe I'm reading it wrong but I cant seem to find any syntax error with my grok filter.
I debugged the GROK Pattern with the heroku debugger and it worked fine. I have also uploaded the log manually to Elasticsearchs Machine Learning tool and tried that grok filter out. Also to no avail.

You match a field against a string. The string has to be in quotes. Since you have double quotes in your pattern you should surround it in single quotes.

match => { "message" => '{"name":%{QUOTEDSTRING:name},"pid":"%{NUMBER:pid}","start_time":"%{NUMBER:date}"}' }
1 Like

Hi Badger, I tried your solution but I'm still getting the same error message.

The only time I was able to successfully boot my logstash instance was when I got rid of the curly brackets so I am guessing Logstash/Grok might have some sort of issue with those.

Try this:
match => { "message" => '{"name":"%{DATA:name}","pid":"%{NUMBER:pid}","start_time":"%{NUMBER:date}"}' }

1 Like

I was able to find a solution for the issue.
I tried to do multiple extractions within the syntax for a single extraction.

My new filter now looks like this:

filter {
  if "osquery" in [tags]{
  grok {
    match => {
      "message" => [
        '\"name\":%{QUOTEDSTRING:name}"',
        '\"pid\":%{NUMBER:pid}"',
        '\"start_time\":"%{NUMBER:date}"'
     ]

Badger and fadjar both were also correct in their usage of single ticks, with the new syntax and single ticks the logstash instance is now booting.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.