Docker Json Logs to haproxy module

Elastic Stack Beats
1

Hi,
My docker logging driver is set to output JSON log files. Changing this setting is out of my control, and means all of the logs in /var/lib/docker/containers/${data.docker.container.id} are in json format. i.e., a log line looks like:

{"log":"172.1.2.124:61930 [09/Apr/2019:11:00:00.043] https~ https/\u003cNOSRV\u003e -1/-1/-1/-1/1 400 0 - - CR-- 1/1/0/0/0 0/0 "\u003cBADREQ\u003e"\n","stream":"stdout","time":"2019-04-09T11:00:00.043787244Z"}

My filebeat.yml configuration is:

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.labels.test: "haproxy-container"
          config:
            - module: haproxy
              log:
                enabled: true
                input:
                  type: docker
                  containers.ids: 
                    - "${data.docker.container.id}"

However, the log message reaches Kibana, and is stored under a json field named 'message', and I don't receive any of the haproxy.* fields from the haproxy module. i.e.,

message 172.1.2.124:61930 [09/Apr/2019:11:00:00.043] https-sni~ https-sni/ -1/-1/-1/-1/2 400 0 - - CR-- 1/1/0/0/0 0/0 ""

I'm guessing I have to tell filebeats to somehow pull the content of the "log" field out of the json, and pass this to the haproxy module, but I'm not sure how to do this. Could someone help?

2

Alternatively, if there's a way to debug this myself I'd also be very grateful for some pointers. I've tried running with "--d autodiscover, haproxy,docker", but it doesn't reveal much. It would be really handy if I could somehow tell whether the haproxy module was even getting invoked?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.