Autodiscover AND extract field defined in module

[Kaj_Noppen]

Hi all,

I have been playing around with filebeat on my CentOS machine. I am trying to get filebeat on the host OS to send logs of my Apache)containers to Elasticsearch, whilst also extracting the fields by using the Apache2 module. For some reason I am not able to get this to work. I am able to extract the raw Docker json messages, or nothing at all.

So I'd like to achieve that this message is nicely split in to fields, as defined in Apache2 module.

"message": "{"log":"1.1.1.1 - - [19/Nov/2018:16:00:38 +0000] \"GET /somepage HTTP/1.1\" 200 427 \"https://URL.php\" \"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0\"\n","stream":"stdout","time":"2018-11-19T16:00:38.6229692Z"}",

I have tried various configs, but this seems to be the one I distilled from other topics:

filebeat.autodiscover:
  providers:
   - type: docker
     templates:
       - condition.contains:
           docker.container.image: website
         config:
          - module: apache2
            error:
              paths:
                - "/docker/containers/${data.docker.container.id}/*.log"

I am running filebeat 6.5.0 with docker 18.06.1-ce. Docker is running under /docker/, not in /var/lib/docker. Filebeat is not running in a container itself.

[Kaj_Noppen]

Also tried an alternative setup:

filebeat.autodiscover:
providers:
- type: docker
templates:
- condition.contains:
docker.container.image: website
config:
- module: apache2
error:
type: docker
paths:
- /docker/containers/${data.docker.container.id}/*.log
containers.ids:
- ${data.kubernetes.container.id}

gives the error:

ERROR [autodiscover] cfgfile/list.go:96 Error creating runner from config: No paths were defined for input accessing config

[Kaj_Noppen]

In the end found another blog post which had a working config:

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.image: website
          config:
            - module: apache2
              access:
                enabled: true
                input:
                  type: docker
                  containers.ids:
                    - "${data.docker.container.id}"
              error:
                enabled: true
                var.hosts: ["${data.host}:${data.port}"]

Also, I needed to refresh my mappings in Kibana. This because the Apache fields had not been loaded in there yet.. (doh!)