Apache Docker containers autodiscovery ends up with grok error for error messages

I set up a new instance of Elasticsearch, Kibana, and various beats today. Since my sites are all in containers, I eventually found my way to the autodiscovery settings.

I've been able to configure access logs so that Filebeat processes them just fine.

But since docker puts stderr into the same file as stdout, Filebeat tries to treat access logs as error logs if the error log config is last, or error logs as access logs if the access log config is last (or the only config.)

You end up with error.message fields like this:

Provided Grok expressions do not match field value: [[Mon Mar 01 03:28:30.532028 2021] [core:crit] [pid 29] (13)Permission denied: [client 10.5.1.1:47622] AH00529: /var/www/html/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable and that '/var/www/html/' is executable, referer: https://davidreagan.net/biography]

(I manually messed up permissions on the site.)

Or the normal restart message:

Provided Grok expressions do not match field value: [[Mon Mar 01 03:30:14.749871 2021] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND']

In the docker log file it looks like:

{"log":"[Mon Mar 01 03:30:14.749871 2021] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'\n","stream":"stderr","time":"2021-03-01T03:30:14.750006026Z"}

My autodiscover settings look something like:

autodiscover:
  providers:
    - type: docker
      containers.stream: stdout
      templates:
        - condition:
            or:
              - contains:
                  docker.container.image: "alpha"
              - contains:
                  docker.container.image: "beta"
          config:
            - module: apache
              access:
                enabled: true
                var.paths: []
                input:
                  type: container
                  paths:
                    - /var/lib/docker/containers/${data.docker.container.id}/*.log
#    - type: docker
#      containers.stream: stderr
#      templates:
#        - condition:
#            or:
#              - contains:
#                  docker.container.image: "alpha"
#              - contains:
#                  docker.container.image: "beta"
#          config:
#            - module: apache
#              error:
#                enabled: true
#                var.paths: []
#                input:
#                  type: container
#                  paths:
#                    - /var/lib/docker/containers/${data.docker.container.id}/*.log

As you can see I tried to split things with the containers.stream: stderr|stdout, but that didn't seem to do anything, and I didn't see it mentioned as usable like that in the docs.

I have stderr section commented out for now, since you get more access logs than error logs...

I did spend a good while searching for answers and didn't find anything helpful with the search terms I used.

So, anyone want to help?

How can I configure Filebeat so that logs from stdout are treated as access logs, and logs from stderr are treated as error logs?

Thanks in advance!

FYI, I'm on 7.11.1 for all ELK Stack apps.

Anyone? I'm still stuck on this.

Hi, could you remove this line from under autodiscover.providers[0]:

containers.stream: stdout

And add this line under autodiscover.providers[0].templates.config[0].access.input:

stream: stdout

Thanks,

Shaunak

Ok, that helped me limit it to just the stdout logs, and I now have a "stream" field.

But I can't seem to get the stderr logs to work. They aren't showing up at all. The only value for stream is "stdout"

Here are the three things I've tried, each time making sure to generate error logs before checking Kibana.

Adding error config to the same section as the access config.

        autodiscover:
          providers:
            - type: docker
              templates:
                - condition:
                    or:
                      - contains:
                          docker.container.image: "imagea"
                      - contains:
                          docker.container.image: "imageb"
                  config:
                    - module: apache
                      access:
                        enabled: true
                        var.paths: [ ]
                        input:
                          stream: stdout
                          type: container
                          paths:
                            - /var/lib/docker/containers/${data.docker.container.id}/*.log
                      error:
                        enabled: true
                        var.paths: [ ]
                        input:
                          stream: stderr
                          type: container
                          paths:
                            - /var/lib/docker/containers/${data.docker.container.id}/*.log

Adding a second "module" config just for error.

        autodiscover:
          providers:
            - type: docker
              templates:
                - condition:
                    or:
                      - contains:
                          docker.container.image: "imagea"
                      - contains:
                          docker.container.image: "imageb"
                  config:
                    - module: apache
                      access:
                        enabled: true
                        var.paths: [ ]
                        input:
                          stream: stdout
                          type: container
                          paths:
                            - /var/lib/docker/containers/${data.docker.container.id}/*.log
                    - module: apache
                      error:
                        enabled: true
                        var.paths: [ ]
                        input:
                          stream: stderr
                          type: container
                          paths:
                            - /var/lib/docker/containers/${data.docker.container.id}/*.log

Adding a completely seperate providers config just for error.

        autodiscover:
          providers:
            - type: docker
              templates:
                - condition:
                    or:
                      - contains:
                          docker.container.image: "imagea"
                      - contains:
                          docker.container.image: "imageb"
                  config:
                    - module: apache
                      access:
                        enabled: true
                        var.paths: [ ]
                        input:
                          stream: stdout
                          type: container
                          paths:
                            - /var/lib/docker/containers/${data.docker.container.id}/*.log
            - type: docker
              templates:
                - condition:
                    or:
                      - contains:
                          docker.container.image: "imagea"
                      - contains:
                          docker.container.image: "imageb"
                  config:
                    - module: apache
                      error:
                        enabled: true
                        var.paths: [ ]
                        input:
                          stream: stderr
                          type: container
                          paths:
                            - /var/lib/docker/containers/${data.docker.container.id}/*.log

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.