Parsing HTTP logs

a_kasireddy · January 8, 2018, 5:06pm

Hello All,
I'm trying to parse http logs using filebeat and am running into some issues parsing the ip information.
We have the X-FORWARDED-FOR header enabled, which is sending two ip's for some of the calls we get (below is a sample call).

12.234.21.234, 123.34.567.32 - - [08/Jan/2018:00:00:26 -0500] RspTime= 75484 microsecond + "POST /hubcosmosint/DealerAdmin HTTP/1.1" 200 492 d-dummy-101:1234 "-" "-"

below is the json file we have to parse the data :
{
"description": "Parse HTTP Access Logs",
"processors": [
{
"grok" : {
"field" : "message",
"patterns" : [
"%{IPORHOST:client} -.*- \[%{HTTPDATE:ts}\] (?:RspTime\= %{NUMBER:timetaken} microsecond) %{NOTSPACE:connstatus} "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion
})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:%{HOSTNAME:server})(?:\:%{NUMBER:portnumber}) "%{DATA:referer}" "(?:%{DATA:UserAgent})""
]
}
}
]
}

This works fine for the logs which has call only with single ip but for the logs which has calls with two ip's it is not able to parse the data.

I'm using Grok debugger and see that the below line can be used to parse this data with two ip's

(?%{IP}(, %{IP})*)

But, I unable to add this line to the json file without any exceptions.

Can someone please help me with this?

Thanks,
Amar

steffens · January 9, 2018, 12:39pm

Is this nginx logs?

Have a look at the nginx filebeat module: https://github.com/elastic/beats/blob/master/filebeat/module/nginx/access/ingest/default.json#L4

It uses grok to collect the list of IPs into nginx.access.remote_ip_list + the split processor to create an array. The painless script iterates the IP list, to find the first non-local-IP that can be used as nginx.access.remote_ip, which is also used with GeoIP.

a_kasireddy · January 9, 2018, 4:01pm

Thanks for the response, Steffen.

These are custom http logs.

We were able to Parse the below log using a custom GROK pattern ("%{IPORHOST:client} -.*- [%{HTTPDATE:ts}] (?:RspTime= %{NUMBER:timetaken} microsecond) %{NOTSPACE:connstatus} "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion
})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:%{HOSTNAME:server})(?::%{NUMBER:portnumber}) "%{DATA:referer}" "(?:%{DATA:UserAgent})"").

123.34.567.32 - - [08/Jan/2018:00:00:26 -0500] RspTime= 75484 microsecond + "POST /hubcosmosint/DealerAdmin HTTP/1.1" 200 492 d-dummy-101:1234 "-" "-"

Recently there was a change and multiple IP addresses were coming in the first field seperated by commas and filebeat stopped parsing it.

Below is an example of the new log :

12.234.21.234, 123.34.567.32 - - [08/Jan/2018:00:00:26 -0500] RspTime= 75484 microsecond + "POST /hubcosmosint/DealerAdmin HTTP/1.1" 200 492 d-dummy-101:1234 "-" "-"

I'm looking for a grok pattern to parse the above log.

steffens · January 10, 2018, 12:03pm

Please properly format JSON, logs and configs using the </>-Button.

Have a look at the nginx module. Nginx logging can have multiple IPs, just as you have. You just need to copy part of the nginx grok pattern (+ optionally processing).

abhilashusha · January 15, 2018, 8:19pm

Thanks Steffens

Following is what I did

Grok Pattern Working Logs

172.27.81.113, 192.34.56.67 - - [07/Jan/2018:19:00:30 -0500] RspTime= 555 microsecond + "GET / HTTP/1.1" 200 3493 - "-" "-"

GrokPattern Non Working Log

- - [07/Jan/2018:19:00:30 -0500] RspTime= 666 microsecond + "GET / HTTP/1.1" 600 6493 - "-" "-"
    Grok Pattern I have:
    {
    "description": "Parse HTTP Access Logs",
    "processors": [
    {
    "grok" : {
    "field" : "message",
    "patterns" : [
    "%{NOTSPACE:client} %{NOTSPACE:ident} %{NOTSPACE:auth} [%{HTTPDATE:ts}] (?:RspTime= %{NUMBER:timetaken} microsecond) %{NOTSPACE:connstatus} "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?(?:%{HOSTNAME:server})(?::%{NUMBER:portnumber})|-) "(?:%{DATA:referer}|-)" "(?:%{DATA:UserAgent}|-)""
    ],

I tried (?:%{IPORHOST:client}|-) and %{NOTSPACE:client} , still I face issues with parsing the log which has the first field as -

steffens · January 15, 2018, 9:24pm

@abhilashusha I'm not sure if your grok pattern/problem matches the original discussion. The OP did filter out all fields before the HTTP timestamp, while you seam trying to want to parse those + have to deal with IPs. It's a subtle difference, which warrants another discussion.

abhilashusha · January 15, 2018, 9:57pm

The issue I'm having here is when the first field ( IP address) comes as empty( -) . Everything works fine if the first field is a valid IP address or a string of IPS seperated by commas.

abhilashusha · January 15, 2018, 9:59pm

@steffens , we changed the pattern a little . You can refer to the latest one.

steffens · January 16, 2018, 2:02pm

I see.

Can you include the complete grok definition? Please format using the </>-Button at the top of the text box (reading json/configs is easier when formatted).

You can have anything between 0 and N ips. The IPORHOST pattern does not cover all of these.

I guess logs always have a space character after the iplist, which can either be -, IP or IP1, IP2, IP3, ...?

Some sample logs + grok (formatted to use with simulate API) would be helpful. Not having the full picture it's somewhat hard to make any recommendation. Anyways, I would define OPT_ADDRESS_LIST to be (-|IPORHOST(, IPORHOST)*). This regular expression parses either - (the empty list) or a sequence of IPORHOST separated by ,<space>.

Also have the grok pattern defined to be ^%{OPT_ADDRESS_LIST:client}.

There might be other errors in the grok pattern. If I'm unsure I clear my grok pattern and constructed incrementally. E.g. start with patterns: ["^%{OPT_ADDRESS_LIST:client} .*"]. If this works, add the next item to be parsed.

abhilashusha · January 16, 2018, 2:21pm

Thanks for looking into the issue.

{
"description": "Parse HTTP Access Logs",
"processors": [
{
"grok" : {
"field" : "message",
"patterns" : [
"%{NOTSPACE:client} %{NOTSPACE:ident} %{NOTSPACE:auth} [%{HTTPDATE:ts}] (?:RspTime= %{NUMBER:timetaken} microsecond) %{NOTSPACE:connsta
tus} "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?(?:%{
HOSTNAME:server})(?::%{NUMBER:portnumber})|-) "(?:%{DATA:referer}|-)" "(?:%{DATA:UserAgent}|-)""
],
"pattern_definitions": {
"IP_LIST": "(%{IP}(, %{IP})*)"
},
"ignore_missing": true

}
]
}Preformatted text

abhilashusha · January 16, 2018, 2:33pm

the logs to be parsed would be

12.234.21.234, 123.34.567.32 - - [08/Jan/2018:00:00:26 -0500] RspTime= 75484 microsecond + "POST /url1 HTTP/1.1" 200 492 d-dummy-101:1234 "-" "-"

12.234.21.234 - - [08/Jan/2018:00:00:26 -0500] RspTime= 75484 microsecond + "POST /url2 HTTP/1.1" 200 492 d-dummy-101:1234 "-" "-"

- - [08/Jan/2018:00:00:26 -0500] RspTime= 75484 microsecond + "POST /hubcosmosint/DealerAdmin HTTP/1.1" 200 492 d-dummy-101:1234 "-" "-"
    indent preformatted text by 4 spaces

abhilashusha · January 16, 2018, 4:47pm

@steffens . I did try the pattern as you suggested , but that didn't help

{
"description": "Parse HTTP Access Logs",
"processors": [
{
"grok" : {
"field" : "message",
"patterns" : [
"^%{IP_LIST:client} %{NOTSPACE:ident} %{NOTSPACE:auth} [%{HTTPDATE:ts}] (?:RspTime= %{NUMBER:timetaken} microsecond) %{NOTSPACE:connstatus} "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?(?:%{HOSTNAME:server})(?::%{NUMBER:portnumber})|-) "(?:%{DATA:referer}|-)" "(?:%{DATA:UserAgent}|-)""
],
"pattern_definitions": {
"IP_LIST": "-|IPORHOST(, IPORHOST)*)"
},
"ignore_missing": true

}
]
}
indent preformatted text by 4 spaces

steffens · January 18, 2018, 3:24pm

Please use the </>-button to format your text. Or markedown fenced code block.
Indenting your code/logs doesn't help if no formatting is applied.

Have you had a look at the simulate API to run some tests and share code-snippets other users can actually run and modify?

While grok is based on regular expressions, you should not try to mix it with regular expressions (and create too much grouping). The way you configured your pattern I'm not even sure a correct regular expression will be generated...

Testing with simulate API, I found using %{IP} multiple times doesn't work. A workaround is to just capture everything until space without comma.

See (can be run with kibana developer console):

POST _ingest/pipeline/_simulate
{
  "pipeline" : {
    "description": "Parse HTTP Access Logs",
    "processors": [
      {
        "grok": {
          "field" : "message",
          "patterns" : [
            "^%{IP_LIST:ips} %{ANY:rest}"
          ],
          "pattern_definitions": {
            "IP_LIST": "(?:-|(?:%{IPORHOST}(?:, [^\\s]+)*))",
            "ANY": "(?:.*)"
           }
        }
      }
    ]
  },
  "docs" : [
    { "_source": { "message": "12.234.21.234, 123.34.567.32 - - [08/Jan/2018:00:00:26 -0500] RspTime= 75484 microsecond"} },
    { "_source": { "message": "12.234.21.234 - - [08/Jan/2018:00:00:26 -0500] RspTime= 75484 microsecond"} },
    { "_source": { "message": "- - - [08/Jan/2018:00:00:26 -0500] RspTime= 75484 microsecond"} }
  ]
}

This gets you:

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_type": "_type",
        "_id": "_id",
        "_source": {
          "rest": "- - [08/Jan/2018:00:00:26 -0500] RspTime= 75484 microsecond",
          "message": "12.234.21.234, 123.34.567.32 - - [08/Jan/2018:00:00:26 -0500] RspTime= 75484 microsecond",
          "ips": "12.234.21.234, 123.34.567.32"
        },
        "_ingest": {
          "timestamp": "2018-01-18T15:18:45.967Z"
        }
      }
    },
    {
      "doc": {
        "_index": "_index",
        "_type": "_type",
        "_id": "_id",
        "_source": {
          "rest": "- - [08/Jan/2018:00:00:26 -0500] RspTime= 75484 microsecond",
          "message": "12.234.21.234 - - [08/Jan/2018:00:00:26 -0500] RspTime= 75484 microsecond",
          "ips": "12.234.21.234"
        },
        "_ingest": {
          "timestamp": "2018-01-18T15:18:45.967Z"
        }
      }
    },
    {
      "doc": {
        "_index": "_index",
        "_type": "_type",
        "_id": "_id",
        "_source": {
          "rest": "- - [08/Jan/2018:00:00:26 -0500] RspTime= 75484 microsecond",
          "message": "- - - [08/Jan/2018:00:00:26 -0500] RspTime= 75484 microsecond",
          "ips": "-"
        },
        "_ingest": {
          "timestamp": "2018-01-18T15:18:45.967Z"
        }
      }
    }
  ]
}

a_kasireddy · February 1, 2018, 4:50pm

Hello @steffens,
We tried the above pattern to list IP's but still couldn't parse the calls successfully.

We are running on elasticsearch 5.3.0 and using same version of filebeat.
As we see most of the updates online are related to v6.1, we tried to upgrade our filebeat to 6.1.2.

after upgrading to 6.1.2, we are not able to parse the calls any call (even the calls with single IP).

Below is the error I see when parsing :
2018-02-01T11:38:20-05:00 WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"Failed to parse mapping [default]: Mapping definition for [error] has unsupported parameters: [properties : {code={type=long}, message={norms=false, type=text}, type={ignore_above=1024, type=keyword}}]","caused_by":{"type":"mapper_parsing_exception","reason":"Mapping definition for [error] has unsupported parameters: [properties : {code={type=long}, message={norms=false, type=text}, type={ignore_above=1024, type=keyword}}]"}}

I tried to use the same grok pattern we used for 5.3.0 and the same works for 5.3.0 but with 6.1.2, it is not working.

Am I missing something here? Do we need to do any additional configurations on 6.1.2?

Please let us know if you see something is missing here.

Thanks,
Amar

a_kasireddy · February 1, 2018, 5:05pm

I enabled the apache2 module, installed the ingest-geoip and ingest-user-agent plugins as mentioned in the user guide

system · March 1, 2018, 5:05pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Nginx & proxies Beats filebeat	4	1022	June 13, 2017
How to parse filebeat access log with vhost? Beats filebeat	4	1143	June 26, 2018
One line JSON parsing into separate events/logs Beats filebeat	2	423	August 15, 2019
Cant get grok working with nginx log Beats filebeat	6	960	February 20, 2022
Filebeat 7.9.1 parsing http_json data Beats filebeat	10	1437	October 30, 2020

Parsing HTTP logs

Related topics