So far though, I only get "Provided Grok expressions do not match field value" errors with any of the configurations I've tried.
Per HAProxy's docs, the formats are: local, rfc3164, rfc5424, raw, and a couple others that don't contain needed info.
Setting any of those options in the global section, and then tcplog or httplog for tcp or http listen stanzas doesn't result any of the logs getting properly grokked.
Here's my haproxy.cfg file and some example logs for the raw format:
global log stderr format raw daemon debug user haproxy group haproxy defaults log global # How many times should we try to connect ot a backend server before giving up? retries 5 # How long before failing a backend server connection timeout connect 10s # How long should pending data be buffered before it is dropped # When this is low, we see "Aborted connection" messages in Galera's logs. timeout server 600s # How long before closing an inactive client connection timeout client 45s # Use gethostbyname() or getaddrinfo() to get ip address of backend server, then # try the last value in the state file, then just let it be nothing. default-server init-addr libc,last,none resolvers docker parse-resolv-conf listen project-mariadb-cluster option tcplog bind *:3306 mode tcp option mysql-check user haproxy balance source server project-mariadb-1 "project-db-01.dbnet:3306" check inter 5s resolvers docker server project-mariadb-2 "project-db-02.dbnet:3306" check inter 5s resolvers docker server project-mariadb-3 "project-db-03.dbnet:3306" check inter 5s resolvers docker listen stats option httplog bind *:1337 mode http maxconn 10 timeout queue 100s stats enable stats hide-version stats refresh 10s stats show-node stats uri /haproxy?stats
That config results in logs like:
project-project-db-1 | <30>Sep 7 19:31:15 haproxy: 127.0.0.1:49442 [07/Sep/2023:19:31:15.656] stats stats/<STATS> 0/0/0/0/0 200 21950 - - LR-- 1/1/0/0/0 0/0 "GET /haproxy?stats HTTP/1.1"
project-project-db-1 | <30>Sep 7 19:31:09 haproxy: 172.19.0.1:39724 [07/Sep/2023:19:31:08.298] project-mariadb-cluster project-mariadb-cluster/project-mariadb-1 1/0/1396 8202373 -- 1/1/0/0/0 0/0
I'm running HAProxy inside a container managed by docker compose. The image is based on
haproxy:2.6.14 I set up filebeat with label based hinting enabled. Since the docs I see in ES do have a "service.type=haproxy" field, I believe I have it configured ok, but just in case, here are the labels:
co.elastic.logs/enabled: "true" co.elastic.logs/module: "haproxy" co.elastic.logs/module.log.enabled: "true"
The last one I'm not sure I need. I'm still figuring out the hinting stuff.
Anyone have any tips?