I set up a new instance of Elasticsearch, Kibana, and various beats today. Since my sites are all in containers, I eventually found my way to the autodiscovery settings.
I've been able to configure access logs so that Filebeat processes them just fine.
But since docker puts stderr into the same file as stdout, Filebeat tries to treat access logs as error logs if the error log config is last, or error logs as access logs if the access log config is last (or the only config.)
You end up with error.message fields like this:
Provided Grok expressions do not match field value: [[Mon Mar 01 03:28:30.532028 2021] [core:crit] [pid 29] (13)Permission denied: [client 10.5.1.1:47622] AH00529: /var/www/html/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable and that '/var/www/html/' is executable, referer: https://davidreagan.net/biography]
(I manually messed up permissions on the site.)
Or the normal restart message:
Provided Grok expressions do not match field value: [[Mon Mar 01 03:30:14.749871 2021] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND']
In the docker log file it looks like:
{"log":"[Mon Mar 01 03:30:14.749871 2021] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'\n","stream":"stderr","time":"2021-03-01T03:30:14.750006026Z"}
My autodiscover settings look something like:
autodiscover:
providers:
- type: docker
containers.stream: stdout
templates:
- condition:
or:
- contains:
docker.container.image: "alpha"
- contains:
docker.container.image: "beta"
config:
- module: apache
access:
enabled: true
var.paths: []
input:
type: container
paths:
- /var/lib/docker/containers/${data.docker.container.id}/*.log
# - type: docker
# containers.stream: stderr
# templates:
# - condition:
# or:
# - contains:
# docker.container.image: "alpha"
# - contains:
# docker.container.image: "beta"
# config:
# - module: apache
# error:
# enabled: true
# var.paths: []
# input:
# type: container
# paths:
# - /var/lib/docker/containers/${data.docker.container.id}/*.log
As you can see I tried to split things with the containers.stream: stderr|stdout
, but that didn't seem to do anything, and I didn't see it mentioned as usable like that in the docs.
I have stderr section commented out for now, since you get more access logs than error logs...
I did spend a good while searching for answers and didn't find anything helpful with the search terms I used.
So, anyone want to help?
How can I configure Filebeat so that logs from stdout are treated as access logs, and logs from stderr are treated as error logs?
Thanks in advance!
FYI, I'm on 7.11.1 for all ELK Stack apps.