Logstash GROK pattern not accepted

[weed101 (dewee)]

Hello,

I'm having a problem with getting my grok pattern to work within my filter. I've created a test.conf file to simplify and narrow down my problem and I cannot seem to get this to work. I have tried the syntax and examples that found in this forum and the docs with no success. I'm hoping someone can see what or where I am going wrong.

My objective is to parse a location string, represented in degrees minutes seconds, into their appropriate fields and later convert them to decimal degrees. But I am stumped on the first step of parsing out the degree minute second field, in their own fields.

Logstash version: 7.0.0

Tested grock pattern at: https://grokdebug.herokuapp.com/

The message being parsed:

313315.621N 0840411.775W

The grok pattern:

(?<lat_degree>\d{2})(?<lat_min>\d{2})(?<lat_sec>[0-9.]{6})(?<lat_dir>\w{1}) (?<lon_degree>\d{2})(?<lon_min>\d{2})(?<lon_sec>[0-9.]{7})(?<lon_dir>\w{1})

Config file:

input { stdin { } }

filter {
    grok { 
      match => { "message", "%{\d{2}:lat_degree}%{\d{2}:lat_min}%{[0-9.]{6}:lat_sec}%{\w{1}:lat_dir} %{\d{3}:lon_degree}%{\d{2}:lon_min}%{[0-9.]{6}:lon_sec}%{\w{1}:lon_dir}" }
    }
}

output {
  stdout { codec => rubydebug }
}

The error message from logstaash command line

[dewee- config]$ logstash -f test.conf
Sending Logstash logs to /opt/logstash/logstash-current/logs which is now configured via log4j2.properties
[2019-04-29T08:00:34,511][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-04-29T08:00:34,527][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.0.0"}
[2019-04-29T08:00:35,735][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 6, column 27 (byte 70) after filter {\n\n grok { \n match => { "message"", :backtrace=>["/opt/logstash/logstash-7.0.0/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative'", "/opt/logstash/logstash-7.0.0/logstash-core/lib/logstash/compiler.rb:49:in compile_graph'", "/opt/logstash/logstash-7.0.0/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2577:in map'", "/opt/logstash/logstash-7.0.0/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in initialize'", "/opt/logstash/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:23:in initialize'", "/opt/logstash/logstash-7.0.0/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "/opt/logstash/logstash-7.0.0/logstash-core/lib/logstash/agent.rb:325:in block in converge_state'"]}
[2019-04-29T08:00:35,999][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2019-04-29T08:00:40,883][INFO ][logstash.runner ] Logstash shut down.

[Badger]

@weed101

match => { "message", "%{\d{2}:lat_degree}...

That should be

match => { "message" => "%{\d{2}:lat_degree}...

[weed101 (dewee)]

Thank you for the quick response.

I tried, but I get another error.

My update looks like this:

filter {
    grok { 
      match => { "message" => "%{\d{2}:lat_degree}%{\d{2}:lat_min}%{[0-9.]{6}:lat_sec}%{\w{1}:lat_dir} %{\d{3}:lon_degree}%{\d{2}:lon_min}%{[0-9.]{6}:lon_sec}%{\w{1}:lon_dir}" }
    }
}

The messages from my attempt to run is

Sending Logstash logs to /opt/logstash/logstash-current/logs which is now configured via log4j2.properties
[2019-04-29T11:25:16,068][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-04-29T11:25:16,087][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.0.0"}
[2019-04-29T11:25:23,884][ERROR][logstash.javapipeline    ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Grok::PatternError: pattern %{\d{2} not defined>, :backtrace=>["/opt/logstash/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:123:in `block in compile'", "org/jruby/RubyKernel.java:1411:in `loop'", "/opt/logstash/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:93:in `compile'", "/opt/logstash/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:281:in `block in register'", "org/jruby/RubyArray.java:1792:in `each'", "/opt/logstash/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:275:in `block in register'", "org/jruby/RubyHash.java:1419:in `each'", "/opt/logstash/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:270:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:56:in `register'", "/opt/logstash/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:191:in `block in register_plugins'", "org/jruby/RubyArray.java:1792:in `each'", "/opt/logstash/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:190:in `register_plugins'", "/opt/logstash/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:446:in `maybe_setup_out_plugins'", "/opt/logstash/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:203:in `start_workers'", "/opt/logstash/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:145:in `run'", "/opt/logstash/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:104:in `block in start'"], :thread=>"#<Thread:0x556eacc8 run>"}
[2019-04-29T11:25:23,905][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2019-04-29T11:25:24,397][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2019-04-29T11:25:29,123][INFO ][logstash.runner          ] Logstash shut down.

Am I using the wrong syntax of the grok pattern? It doesn't like my pattern for some reason.

[Badger]

grok has a set of patterns that are expanded into regular expression. You can also use custom patterns. So the first two parts of your grok pattern should be

"message" => "^(?<lat_degree>\d{2})(?<lat_min>\d{2})"

I add the ^ to anchor the pattern. If you do not want to match the start of the message you can remove it.

[weed101 (dewee)]

Ahhh.... That worked perfectly. Thank you very much.

I see I was using the wrong syntax like the braces instead of parentheses.