Good afternoon everyone,
I have a server where are running one instance of haproxy and one instance of filebeat in different containers.
In order to have the output of haproxy accesible by docker logs, I have configured the output in stdout as recommended in the HAProxy blog.
defaults
log stdout format raw local0 info
mode http
option httplog
This generates the following output when I execute the docker logs command:
XX.XX.XX.XXX:YYYYY [03/Dec/2020:15:29:08.146] http-in backend/appserver 0/0/0/1/1 200 217 - - ---- 3/3/0/0/0 0/0 "HEAD /inf/test.html HTTP/1.1"
My problem comes, because I have configured my filebeat.yml to work with autodiscovery and the haproxy module:
filebeat:
autodiscover.providers:
- type: docker
templates:
- condition:
contains:
docker.container.image: haproxy
config:
- module: haproxy
log:
input:
type: container
paths:
- /var/lib/docker/containers/${data.docker.container.id}/*.log
But the previous configuration doesn't seem to work, since once I check for the data in Kibana, the entire line is under the message field, and the haproxy fields aren't present.
When I checked the grok in the haproxy-module, I saw that it was expecting the process name and the pid at the beginning of the message field, so I tried modifying the log-format to a custom one following the grok expectations, but it doesn't work neither.
defaults
log stdout format raw local0 info
log-format "haproxy[%pid]: %ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"
mode http
Does someone know how to configure it properly or if is it a bug from the combination of autodiscover + haproxy module?
UPDATE:
In order to check the reason, I also tried to include the apache module as follows, but it doesn't seem either to work.
filebeat:
inputs:
- type: log
enabled: true
paths:
- /var/lib/docker/volumes/monitor_logs/_data/*
autodiscover.providers:
- type: docker
templates:
- condition:
contains:
docker.container.image: haproxy
config:
- module: haproxy
log:
input:
type: container
paths:
- /var/lib/docker/containers/${data.docker.container.id}/*.log
- condition:
contains:
docker.container.image: httpd
config:
- module: apache
access:
input:
type: container
paths:
- /var/lib/docker/containers/${data.docker.container.id}/*.log
I was expecting something like the fields shown in https://www.elastic.co/guide/en/beats/filebeat/current/exported-fields-apache.html but it still just adds the whole log line under the message field:
