[HAproxy] log pipeline fails to extract http.request.method for HTTP/2.0 requests

Mydayyy · July 5, 2022, 7:41pm

Greetings,

in accordance with your Github guideline, I am reporting this issue here.

Please post all questions and issues on Beats - Discuss the Elastic Stack
before opening a Github Issue. Your questions will reach a wider audience there,
and if we confirm that there is a bug, then you can open a new issue.

The HAproxy Log parser implementation fails to parse the http.request.method with HAproxy log lines generated by HTTP/2.0 requests. The reason behind that is, that HAproxy logs the full URL for requests made with HTTP/2.0 as opposed to only the path as it was in HTTP/1.1.

The grok expression can be found here: https://github.com/elastic/beats/blob/master/filebeat/module/haproxy/log/ingest/pipeline.yml#L63

%{WORD:http.request.method}%{SPACE}%{URIPATHPARAM:url.original}%{SPACE}HTTP/%{NUMBER:http.version}

Executing that expression against a HTTP/1.1 logline works as expected:
Line

POST /credentials HTTP/1.1

Result:

{
  "http": {
    "request": {
      "method": "POST"
    },
    "version": "1.1"
  },
  "url": {
    "original": "/credentials"
  }
}

Executing that expression against a HTTP/2.0 logline delivers wrong results:
Line:

GET https://subdomain.domain.tld/test HTTP/2.0

Result:

{
  "http": {
    "request": {
      "method": "tld"
    },
    "version": "2.0"
  },
  "url": {
    "original": "/test"
  }
}

(Note the method was extracted as tld)

Relevant HAproxy issue: Log-format %HU for http/2 requests logs full url (including protocol) - Help! - HAProxy community

Note: This seems to be an intended change for HTTP/2.0

The implementation for the parsing http.request.method was originally discussed here:

github.com/elastic/beats

[Filebeat] [HaProxy] Parse http and url fields

opened 11:25AM - 01 May 21 UTC

closed 02:00PM - 03 May 21 UTC

SHolzhauer

needs_team

**Describe the enhancement:** The current pipeline for HaProxy does not parse t…he `haproxy.http.request.raw_request_line` further into possible fields. Example: ``` { "haproxy.http.request.raw_request_line": "POST /path/to/page HTTP/1.1" } ``` Can be further parsed into: ``` { "haproxy": { "http": { "request": { "raw_request_line": "POST /path/to/page HTTP/1.1" } } }, "http": { "request": { "method": "POST" ", "version": "1.1" }, "url": { "path": "/path/to/page" } } ``` I'm not entirely sure if HaProxy logs request parameters (query) but if so the path can be split into it ofcourse: ``` { "haproxy.http.request.raw_request_line": "POST /path/to/page?key=value HTTP/1.1" } ``` Can be further parsed into: ``` { "haproxy": { "http": { "request": { "raw_request_line": "POST /path/to/page?key=value HTTP/1.1" } } }, "http": { "request": { "method": "POST" ", "version": "1.1" }, "url": { "path": "/path/to/page", "query": "key=value" } } ``` **Describe a specific use case for the enhancement or feature:** More complete parsing will allow for better monitoring in both observability (how often is page `y` opened) as well as security.

Best Regards
Mydayyy

Note:

I fixed it by adding another pattern for the full url before the pattern from above:

      "grok": {
        "field": "haproxy.http.request.raw_request_line",
        "ignore_missing": true,
        "patterns": [
          "%{WORD:http.request.method}%{SPACE}%{URI:url.original}%{SPACE}HTTP/%{NUMBER:http.version}",
          "%{WORD:http.request.method}%{SPACE}%{URIPATHPARAM:url.original}%{SPACE}HTTP/%{NUMBER:http.version}"
        ]
      }
    },

jsoriano · July 6, 2022, 9:06am

Hey @Mydayyy, welcome to discuss and thanks a lot for reporting this problem! I have added it to this issue with a list of problematic logs with current haproxy integration: Increase support of log formats in haproxy filebeat module · Issue #3250 · elastic/integrations · GitHub

system · August 3, 2022, 11:06am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.