How to parse Haproxy log with grok

Elastic Stack Logstash
1

Hello,

I am a new elastic stack user.

I want to parse the following log :

Dec 3 14:05:17 f774bd25748a local0.info haproxy[1]: unix:1 [03/Dec/2018:14:05:17.559] frontend.https.services~ https.services.tac/https.services.tac.1 0/0/1/13/14 200 1058 - - ---- 1773/140/0/0/0 0/0 "POST /tac/medias/access HTTP/1.1"Dec 3 14:05:17 10.0.0.3 frontend_proxy.qed9mwptyw6kdo3i9y9uogdl0.jie6pdr[15900]: Dec 3 14:05:17 f6047fe55160 local0.info haproxy[1]: unix:1 [03/Dec/2018:14:05:17.506] frontend.https.services~ https.services.css/https.services.css.2 0/0/1/70/72 200 1673 - - ---- 1768/145/0/0/0 0/0 "POST /css/catalog/medias/d55c1a3e-1c90-4a8c-8c21-dc23fdab4024?bundleItemT

I tried with the following filter from the dev tools console of elasticsearch but it doesn't work :

(?:%{SYSLOGTIMESTAMP:syslog_timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) %{IPORHOST:syslog_server} %{SYSLOGPROG}: %{IP:client_ip}:%{INT:client_port} [%{HAPROXYDATE:accept_date}] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{INT:time_queue}/%{INT:time_backend_connect}/%{NOTSPACE:time_duration} %{NOTSPACE:bytes_read} %{NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/%{INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %{INT:srv_queue}/%{INT:backend_queue}

Thank you for your help.

2

Try re-creating your grok filter from scratch again using: https://grokdebug.herokuapp.com/
The example you pasted and the grok filter don't match up at all... it seems to fall apart after:
%{IPORHOST:syslog_server}

3

Here my solution.

%{SYSLOGTIMESTAMP:timestamp} %{DATA:Aloha_name} %{DATA:SyslogFacility} %{DATA:Haproxy_process}: %{DATA:client_ip}:%{INT:client_port} \[%{HAPROXYDATE:accept_date}\] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{INT:Tq}/%{INT:Tw}/%{INT:Tc}/%{INT:Tr}/%{INT:Tt} %{INT:http_status_code} %{NOTSPACE:bytes_read} - - ---- %{INT:actconn}/%{INT:feconn}/%{INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %{INT:srv_queue}/%{INT:backend_queue} "%{WORD:Method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}"

https://cdn.haproxy.com/wp-content/uploads/2017/07/aloha_load_balancer_memo_log.pdf

4

That's what I did :wink:

5

If you are using Filebeat to collect the logs, then you can enable the haproxy module that does the parsing for you, without the need to write your own grok filter.

6

I don't think I can do that because my logs are prefixed by syslog.

syslog + haproxy log

Dec  4 15:21:02 10.0.0.3 frontend_proxy.qed9mwptyw6kdo3i9y9uogdl0.jie6pdr[15900]: Dec  4 15:21:02 f6047fe55160 local0.info haproxy[1]: unix:1 [04/Dec/2018:15:21:02.563] frontend.https.services~ https.services.cmm/https.services.cmm.2 0/0/1/18/20 200 3920 - - ---- 1968/186/0/0/0 0/0 "GET /cmm/customers/b296a899-0421-4e00-9bf8-f5e7331c6a9a HTTP/1.1"

I don't have access to the host machine

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.