Kibana Ingest Pipeline Parsing exception

Elastic Stack Logstash
1

I'm trying to ingest an HTTP log through file beat into kibana Pipeline and getting the parse error. The Grok Pattern worked well with grok debugger but the same pattern failed when I attempt to ingest through kibana pipeline!

Sample Log file:

10.199.19.191 - - [14/May/2018:22:47:13 -0700] "POST /cleanse/ HTTP/1.1" 200 4
10.199.19.191 - - [14/May/2018:22:47:43 -0700] "POST /cleanse/ HTTP/1.1" 200 4
10.199.19.191 - - [14/May/2018:22:48:13 -0700] "POST /cleanse/ HTTP/1.1" 200 4
10.199.19.191 - - [14/May/2018:22:48:43 -0700] "POST /cleanse/ HTTP/1.1" 200 4

Grok Pattern:

%{IPORHOST:clientip} (?:-|%{USER:ident}) (?:-|%{USER:auth}) [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|-)" %{NUMBER:response} (?:-|%{NUMBER:bytes})

Snag_5428123

Snag_54ae6db

3

Looks like you have a \ in front of a [ in your pattern, and it is not needed.

4

Thanks Bill , removing \ in front of %{HTTPDATE:timestamp} didn't solve completely until i remove it after too. Though the parsing error is no longer an issue, my logs are rejecting with following from my filebeat logs. Can you please advise on how i can make this work this pattern through ingest pipeline.

2018-06-01T00:31:48Z DBG Bulk item insert failed (i=3, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Provided Grok expressions do not match field value: [10.194.38.218 - - [20/Feb/2018:22:07:08 +0000] "GET /favicon.ico HTTP/1.1" 404 1167]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: Provided Grok expressions do not match field value: [10.194.38.218 - - [20/Feb/2018:22:07:08 +0000] "GET /favicon.ico HTTP/1.1" 404 1167]","caused_by":{"type":"illegal_argument_exception","reason":"Provided Grok expressions do not match field value: [10.194.38.218 - - [20/Feb/2018:22:07:08 +0000] "GET /favicon.ico HTTP/1.1" 404 1167]"}},"header":{"processor_type":"grok"}}
2018-06-01T00:31:48Z DBG Bulk item insert failed (i=4, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Provided Grok expressions do not match field value: [10.194.38.218 - - [20/Feb/2018:22:07:08 +0000] "GET /favicon.ico HTTP/1.1" 404 1168]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: Provided Grok expressions do not match field value: [10.194.38.218 - - [20/Feb/2018:22:07:08 +0000] "GET /favicon.ico HTTP/1.1" 404 1168]","caused_by":{"type":"illegal_argument_exception","reason":"Provided Grok expressions do not match field value: [10.194.38.218 - - [20/Feb/2018:22:07:08 +0000] "GET /favicon.ico HTTP/1.1" 404 1168]"}},"header":{"processor_type":"grok"}}

Here is the screenshot of my pipeline

Snag_8f22f99

5

I'm not too familiar with Grok unfortunately. I would suggest asking in the Logstash forum as they might be able to give you a better answer.

6

Thanks, I moved this under Logstash Forum and hoping that someone would help me on this.

7

Why not use one of the predefined grok patterns for HTTP logs? Your log looks pretty standard. Even if the predefined patterns don't work out of the box it should be quite easy to adapt them to fit your particular log.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.