Grok debugger in dev tools parses correctly, fails running logstash

[starrynight]

I was attempting to parse an atlassian bitbucket log here -- with the log example being this:

10.10.10.2,10.11.1.2 | https | i@5EB8GFx642x581467x0 | - | 2019-08-27 10:42:28,325 | "GET /security_logout HTTP/1.1" | "" "Zabbix" | - | - | - | - | - | 18oho66 |

...which ended up coming up with the following grok statement using the dev tools grok debugger in kibana (on ver 7.3.0) and showed as correct:

%{IPV4:client_ip},%{IPV4:proxy_ip} \| %{DATA:protocol} \| %{DATA:request_id} \| %{HTTPDUSER:user} \| %{TIMESTAMP_ISO8601:datelog} \| "%{DATA:action} %{DATA:resource} %{DATA:http_version}" \| "" "%{DATA:request_details}" \| %{DATA:response} \| %{DATA:bytes_read} \| %{DATA:bytes_written} \| %{DATA:labels} \| %{DATA:response_time} \| %{DATA:session_id} \|

Upon adding it in the correct format in logstash conf.d dir and restarting it fails on compile and on checking it on the grok debugger at herokuapp.com, it is showing with a compile error as well. Confused...

[Badger]

Please edit your post, select the log message and click on </> in the toolbar above the edit pane. Make sure that the appearence of the log message changes to be

like this

Then do the same for the grok pattern.

[starrynight]

I've done this, thanks

[Badger]

What error message does logstash output? That correctly parses the input line for me.

[starrynight]

This is what I'm getting:

Sep 18 13:57:20 hostname logstash[22451]: [2019-09-18T13:57:20,972][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 106, column 166 (byte 2739) after filter {\n if [id] == [\"atlbbaccess\"] {\n grok {\n match => {\"message\" => \"^%{IPV4:client_ip},%{IPV4:proxy_ip} \\| %{DATA:protocol} \\| %{DATA:request_id} \\| %{HTTPDUSER:user} \\| %{TIMESTAMP_ISO8601:datelog} \\| \"", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:incompile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:inblock in compile_sources'", "org/jruby/RubyArray.java:2577:in map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:incompile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in block in converge_state'"]}

[Badger]

@starrynight

%{TIMESTAMP_ISO8601:datelog} | "%{DATA:action}

You need to escape the double quotes, or, as I did, use single quotes around the grok pattern.