Unable to Parse HTTP Logs when the remote IP is empty

Elastic Stack Beats
1

HTTP Parsing fails

Grok Pattern Working Logs

172.27.81.113, 192.34.56.67 - - [07/Jan/2018:19:00:30 -0500] RspTime= 555 microsecond + "GET / HTTP/1.1" 200 3493 - "-" "-"

GrokPattern Non Working Log

- - - [07/Jan/2018:19:00:30 -0500] RspTime= 666 microsecond + "GET / HTTP/1.1" 600 6493 - "-" "-"

Grok Pattern I have:

{
"description": "Parse HTTP Access Logs",
"processors": [
  {
        "grok" : {
      "field" : "message",
      "patterns" : [
       "%{NOTSPACE:client} %{NOTSPACE:ident} %{NOTSPACE:auth} \[%{HTTPDATE:ts}\] (?:RspTime\= %{NUMBER:timetaken} microsecond) %{NOTSPACE:connstatus} \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?(?:%{HOSTNAME:server})(?:\:%{NUMBER:portnumber})|-) \"(?:%{DATA:referer}|-)\" \"(?:%{DATA:UserAgent}|-)\""
                      ],

I tried (?:%{IPORHOST:client}|-) and %{NOTSPACE:client}, still I face issues with parsing the log which has the first field as -.

Can anyone help us?

2

The non working logs is first three values are -.

- - - [07/Jan/2018:19:00:30 -0500] RspTime= 666 microsecond + "GET / HTTP/1.1" 600 6493 - "-" "-"
3

You can specify multiple patterns and grok will try each one. patterns accepts a list. So if it's easier for you to handle each of these logs as two separate grok patterns then do this.

Are you aware of the simulate API for testing ingest node and the grok tester in Kibana.

Also you might find look it useful to look at the ingest pipeline used by the nginx module. https://github.com/elastic/beats/blob/59f728a60239d5464575beef911b2ee9a9f2427e/filebeat/module/nginx/access/ingest/default.json#L7

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.