Custom Pattern in grok filter

Elastic Stack Logstash
1

Hi, I am very new to ELK. I was trying to apply inline custom pattern to my log file but my approach is giving me errors. Can someone help me on this?

template of logs:
172.31.29.134 - - [02/Feb/2018:06:25:05 +0000] "GET /favicon.ico HTTP/1.1" 200 1150 "-" "ELB-HealthChecker/2.0"
172.31.30.6 - - [02/Feb/2018:06:25:06 +0000] "GET / HTTP/1.1" 200 82550 "-" "ELB-HealthChecker/2.0"
172.31.10.17 - - [02/Feb/2018:06:25:07 +0000] "GET /favicon.ico HTTP/1.1" 200 1150 "-" "ELB-HealthChecker/2.0"
172.31.28.216 - - [02/Feb/2018:06:25:08 +0000] "GET /favicon.ico HTTP/1.1" 200 1150 "-" "ELB-HealthChecker/2.0"
172.31.4.52 - - [02/Feb/2018:06:25:08 +0000] "GET / HTTP/1.1" 200 82550 "-" "ELB-HealthChecker/2.0"
172.31.29.66 - - [02/Feb/2018:06:25:09 +0000] "GET /favicon.ico HTTP/1.1" 200 1150 "-" "ELB-HealthChecker/2.0"
172.31.30.6 - - [02/Feb/2018:06:25:12 +0000] "GET /c/oyo?id=IXCP3614&amount=16800&param1=-1 HTTP/1.1" 204 0 "https://www.oyorooms.com/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
172.31.12.194 - - [02/Feb/2018:06:25:15 +0000] "GET / HTTP/1.1" 200 82550 "-" "ELB-HealthChecker/2.0"
172.31.0.51 - - [02/Feb/2018:06:25:15 +0000] "GET /favicon.ico HTTP/1.1" 200 1150 "-" "ELB-HealthChecker/2.0"
172.31.20.253 - - [02/Feb/2018:06:25:22 +0000] "GET /favicon.ico HTTP/1.1" 200 1150 "-" "ELB-HealthChecker/2.0"

grok filter :
grok {
match => { "message" => "(?([0-9]{1,3}.)[0-9]{1,3}) (?[-]) (?[-]) [(?[0-9]{1,2}/(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)/[0-9]{1,4}:[0-9]{1,2}:[0-9]{1,2}:[0-9]{1,2}\s+[0-9]{1,4})] "(?GET|PUT|POST|PATCH) (?/[a-zA-Z0-9]+/[a-zA-Z0-9]+)?(?([a-zA-Z0-9]+=[a-zA-Z0-9]+&)[a-zA-Z0-9]+=[a-zA-Z0-9]+) HTTP/(?[0-9]+.[0-9])" (?[0-9]+) (?[0-9]+) "(?(http(s)?://)?(w.)?([a-zA-Z0-9]+.)+[a-zA-Z0-9]+(((?)?/?=?#?:?&?.?\s?[a-zA-Z0-9]?))?)" "(?.*)""}
}

2

This looks like a pretty typical apache/nginx log file, why are you building a pattern from scratch?

3

Learning purpose. In case there is a unique pattern in future, I would be able to handle it.
I just want to know what is the syntax error. How can I include the custom pattern in grok filter?

4

You haven't provided the error so we can't really tell.

5

grok {
match => { "message" => "(?([0-9]{1,3}.)[0-9]{1,3}) (?[-]) (?[-]) [(?[0-9]{1,2}/(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)/[0-9]{1,4}:[0-9]{1,2}:[0-9]{1,2}:[0-9]{1,2}\s+[0-9]{1,4})] "(?GET|PUT|POST|PATCH) (?/[a-zA-Z0-9]+/[a-zA-Z0-9]+)?(?([a-zA-Z0-9]+=[a-zA-Z0-9]+&)[a-zA-Z0-9]+=[a-zA-Z0-9]+) HTTP/(?[0-9]+.[0-9])" (?[0-9]+) (?[0-9]+) "(?(http(s)?://)?(w.)?([a-zA-Z0-9]+.)+[a-zA-Z0-9]+(((?)?/?=?#?:?&?.?\s?[a-zA-Z0-9]?))?)" "(?.*)""}
}

Error:
[2018-04-25T12:17:33,989][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 9, column 233 (byte 289) after filter {\n \tgrok {\n match => { "message" => "(?([0-9]{1,3}\.)*[0-9]{1,3}) (?[-]) (?[-]) \[(?[0-9]{1,2}\/(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\/[0-9]{1,4}\:[0-9]{1,2}\:[0-9]{1,2}\:[0-9]{1,2}\s\+[0-9]{1,4})] "", :backtrace=>["/home/akash/logstash-6.2.3/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/home/akash/logstash-6.2.3/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/home/akash/logstash-6.2.3/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/home/akash/logstash-6.2.3/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/home/akash/logstash-6.2.3/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "/home/akash/logstash-6.2.3/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "/home/akash/logstash-6.2.3/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/home/akash/logstash-6.2.3/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "/home/akash/logstash-6.2.3/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/home/akash/logstash-6.2.3/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/home/akash/logstash-6.2.3/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "/home/akash/logstash-6.2.3/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/home/akash/logstash-6.2.3/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/home/akash/logstash-6.2.3/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/home/akash/logstash-6.2.3/logstash-core/lib/logstash/agent.rb:90:in execute'", "/home/akash/logstash-6.2.3/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "/home/akash/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

6

You are using unescaped " within a text that is wrapped in ". LogStash thinks that the pattern ends there and doesn't know why more text is following. I think it should work if you wrap the pattern with ' instead.

7

@Akash_Tanwar
For when you give up, we have some examples here to get you going: https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns

8

Thank You! I just put all the special characters like spaces, quotes and forward slashes with a '' and not as it is. And that solved my problem.

Thank you all for helping a newbie :slight_smile:

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.