Kibana Ingest Pipeline Parsing exception

[murali_nutakkivikjuv (Murali Nutakkivikjuvras)]

I'm trying to ingest an HTTP log through file beat into kibana Pipeline and getting the parse error. The Grok Pattern worked well with grok debugger but the same pattern failed when I attempt to ingest through kibana pipeline!

Sample Log file:

10.199.19.191 - - [14/May/2018:22:47:13 -0700] "POST /cleanse/ HTTP/1.1" 200 4
10.199.19.191 - - [14/May/2018:22:47:43 -0700] "POST /cleanse/ HTTP/1.1" 200 4
10.199.19.191 - - [14/May/2018:22:48:13 -0700] "POST /cleanse/ HTTP/1.1" 200 4
10.199.19.191 - - [14/May/2018:22:48:43 -0700] "POST /cleanse/ HTTP/1.1" 200 4

Grok Pattern:

%{IPORHOST:clientip} (?:-|%{USER:ident}) (?:-|%{USER:auth}) [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|-)" %{NUMBER:response} (?:-|%{NUMBER:bytes})

[Bill_McConaghy]

Looks like you have a \ in front of a [ in your pattern, and it is not needed.

[murali_nutakkivikjuv (Murali Nutakkivikjuvras)]

Thanks Bill , removing \ in front of %{HTTPDATE:timestamp} didn't solve completely until i remove it after too. Though the parsing error is no longer an issue, my logs are rejecting with following from my filebeat logs. Can you please advise on how i can make this work this pattern through ingest pipeline.

2018-06-01T00:31:48Z DBG Bulk item insert failed (i=3, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Provided Grok expressions do not match field value: [10.194.38.218 - - [20/Feb/2018:22:07:08 +0000] "GET /favicon.ico HTTP/1.1" 404 1167]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: Provided Grok expressions do not match field value: [10.194.38.218 - - [20/Feb/2018:22:07:08 +0000] "GET /favicon.ico HTTP/1.1" 404 1167]","caused_by":{"type":"illegal_argument_exception","reason":"Provided Grok expressions do not match field value: [10.194.38.218 - - [20/Feb/2018:22:07:08 +0000] "GET /favicon.ico HTTP/1.1" 404 1167]"}},"header":{"processor_type":"grok"}}
2018-06-01T00:31:48Z DBG Bulk item insert failed (i=4, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Provided Grok expressions do not match field value: [10.194.38.218 - - [20/Feb/2018:22:07:08 +0000] "GET /favicon.ico HTTP/1.1" 404 1168]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: Provided Grok expressions do not match field value: [10.194.38.218 - - [20/Feb/2018:22:07:08 +0000] "GET /favicon.ico HTTP/1.1" 404 1168]","caused_by":{"type":"illegal_argument_exception","reason":"Provided Grok expressions do not match field value: [10.194.38.218 - - [20/Feb/2018:22:07:08 +0000] "GET /favicon.ico HTTP/1.1" 404 1168]"}},"header":{"processor_type":"grok"}}

Here is the screenshot of my pipeline

[Bill_McConaghy]

I'm not too familiar with Grok unfortunately. I would suggest asking in the Logstash forum as they might be able to give you a better answer.

[murali_nutakkivikjuv (Murali Nutakkivikjuvras)]

Thanks, I moved this under Logstash Forum and hoping that someone would help me on this.

[magnusbaeck (Magnus Bäck)]

Why not use one of the predefined grok patterns for HTTP logs? Your log looks pretty standard. Even if the predefined patterns don't work out of the box it should be quite easy to adapt them to fit your particular log.