Help with grok pattern for haproxy

Elastic Stack Logstash
1

Hi,please assit with creating custom grok pattern for logstash.

I've created a custom log format in haproxy conf for purpose of logging mutual ssl authentication details for client certificates.
log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %TR/%Tw/%Tc/%Tr/%Ta\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ {%[ssl_c_verify],%{+Q}[ssl_c_s_dn],%{+Q}[ssl_c_i_dn]}\ %{+Q}r\ %sslv\ %sslc

Log line example looks like this:
Jan 17 08:47:06 localhost haproxy[9870]: 10.251.121.117:10300 [17/Jan/2020:08:47:06.531] api.example.com~ test-backend/server1 0/0/1/8/9 201 589 - - ---- 421/419/1/1/0 0/0 {||10.105.74.60} {331260} {0,"/C=GB/L=Tester/O=Some Company That Connects Co./CN=api.somecompany.com","/C=US/O=CertAuth, Inc./OU=See www.certauth.net/legal-terms/OU=(c) 2012 CertAuth, Inc. - for authorized use only/CN=CertAuth Certification Authority - K1B"} "POST /some/1/path HTTP/1.1" TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256

When using standard haproxy grok pattern with additions for tls version and cipher:
%{SYSLOGTIMESTAMP:syslog_timestamp} %{IPORHOST:syslog_server} %{SYSLOGPROG}: %{IP:client_ip}:%{INT:client_port} \[%{HAPROXYDATE:accept_date}\] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{INT:time_request}/%{INT:time_queue}/%{INT:time_backend_connect}/%{INT:time_backend_response}/%{NOTSPACE:time_duration} %{INT:http_status_code} %{NOTSPACE:bytes_read} %{DATA:captured_request_cookie} %{DATA:captured_response_cookie} %{NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/%{INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %{INT:srv_queue}/%{INT:backend_queue} (\{%{HAPROXYCAPTUREDREQUESTHEADERS}\})?( )?(\{%{HAPROXYCAPTUREDRESPONSEHEADERS}\})?( )?"(<BADREQ>|(%{WORD:http_verb} (%{URIPROTO:http_proto}://)?(?:%{USER:http_user}(?::[^@]*)?@)?(?:%{URIHOST:http_host})?(?:%{URIPATHPARAM:http_request})?( HTTP/%{NUMBER:http_version})?))?"( %{NOTSPACE:tls_version})?( %{NOTSPACE:tls_ciphersuite})?

Information from log line that I want to parse ends up in captured_response_headers:

      "HAPROXYCAPTUREDRESPONSEHEADERS": [
        [
          "331260} {0,"/C=GB/L=Tester/O=Some Company That Connects Co./CN=api.somecompany.com","/C=US/O=CertAuth, Inc./OU=See www.certauth.net/legal-terms/OU=(c) 2012 CertAuth, Inc. - for authorized use only/CN=CertAuth Certification Authority - K1B""
        ]
      ],
      "captured_response_headers": [
        [
          "331260} {0,"/C=GB/L=Tester/O=Some Company That Connects Co./CN=api.somecompany.com","/C=US/O=CertAuth, Inc./OU=See www.certauth.net/legal-terms/OU=(c) 2012 CertAuth, Inc. - for authorized use only/CN=CertAuth Certification Authority - K1B""
        ]
      ]

I need to parse captured_response_headers so search is more easier like this:

"id": [
    [
      "331260"
    ]
  ],
  "ssl_verified": [
    [
      "0"
    ]
  ],
  "cert_details": [
    [
      "/C=GB/L=Tester/O=Some Company That Connects Co./CN=api.somecompany.com","/C=US/O=CertAuth, Inc./OU=See www.certauth.net/legal-terms/OU=(c) 2012 CertAuth, Inc. - for authorized use only/CN=CertAuth Certification Authority - K1B""
    ]
  ]
2

Ok, solved it myself.
For posterity that should find this thread in the future:

Default {%{HAPROXYCAPTUREDREQUESTHEADERS}\}
Should be replaced by {%{INT:id}} {%{INT:ssl_verified},%{DATA:client_cert_details},%{DATA:cert_ca}}

The whole grok pattern should look like this:

%{SYSLOGTIMESTAMP:syslog_timestamp} %{IPORHOST:syslog_server} %{SYSLOGPROG}: %{IP:client_ip}:%{INT:client_port} [%{HAPROXYDATE:accept_date}] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{INT:time_request}/%{INT:time_queue}/%{INT:time_backend_connect}/%{INT:time_backend_response}/%{NOTSPACE:time_duration} %{INT:http_status_code} %{NOTSPACE:bytes_read} %{DATA:captured_request_cookie} %{DATA:captured_response_cookie} %{NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/%{INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %{INT:srv_queue}/%{INT:backend_queue} ({%{HAPROXYCAPTUREDREQUESTHEADERS}})?( )?({%{INT:chid}} {%{INT:ssl_verified},%{DATA:client_cert_details},%{DATA:cert_ca}})?( )?"(|(%{WORD:http_verb} (%{URIPROTO:http_proto}://)?(?:%{USER:http_user}(?::[^@]*)?@)?(?:%{URIHOST:http_host})?(?:%{URIPATHPARAM:http_request})?( HTTP/%{NUMBER:http_version})?))?"( %{NOTSPACE:tls_version})?( %{NOTSPACE:tls_ciphersuite})?

3

too soon.
It works on https://grokdebug.herokuapp.com/
But not on logstash-5.6.16-1.noarch it doesn't work.
I'll try latest version

4

Here is final solution for anyone who should find this
%{SYSLOGTIMESTAMP:syslog_timestamp} %{IPORHOST:syslog_server} %{SYSLOGPROG}: %{IP:client_ip}:%{INT:client_port} \[%{HAPROXYDATE:accept_date}\] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{INT:time_request}/%{INT:time_queue}/%{INT:time_backend_connect}/%{INT:time_backend_response}/%{NOTSPACE:time_duration} %{INT:http_status_code} %{NOTSPACE:bytes_read} %{DATA:captured_request_cookie} %{DATA:captured_response_cookie} %{NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/%{INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %{INT:srv_queue}/%{INT:backend_queue} (\{(%{DATA:request_header_x_forwarded_host})?\|(%{DATA:request_header_x_forwarded_for})?\|(%{DATA:request_header_host})?\})?\s?(\{(%{INT:id})?\})?\s?(\{(%{NOTSPACE:cert_verified})?,("%{DATA:client_cert_details}")?,("%{DATA:cert_ca}")?\}( )?")?(<BADREQ>|(%{WORD:http_verb} (%{URIPROTO:http_proto}://)?(?:%{USER:http_user}(?::[^@]*)?@)?(?:%{URIHOST:http_host})?(?:%{URIPATHPARAM:http_request})?( HTTP/%{NUMBER:http_version})?))?"( %{NOTSPACE:tls_version})?( %{NOTSPACE:tls_ciphersuite})?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.