We are using filebeat 7.11.0 along with the postgresql module. However for some reason, the message attribute seems not to be touched, although some metadata seems to be added by the module.
Our log messages look to be the same format as the one used in the tests but don't result in the same expected output.
The logs get sent from filebeat to our logstash, where we see the same log message arrive as the one sent by filebeat in the filebeat debug log.
In the filebeat logs, it seems like it finds the postgresql.yml configuration file and also picks up new logs added to the postgresql file, but the event seems to be published without a parsed message.
For example, this is the line in the log file (names of tables etc. masked):
2021-03-16 13:35:12.231 UTC [37] LOG: statement: SELECT "x"."y", "x"."z", "x"."expire_date" FROM "x" WHERE ("x"."expire_date" > '2021-03-16T13:35:12.190372+00:00'::timestamptz AND "x"."y" = 'xyz') LIMIT 21
And this is the result if I turn debug logging on in filebeat:
2021-03-16T15:02:31.269Z DEBUG [processors] processing/processors.go:203 Publish event: {
"@timestamp": "2021-03-16T15:02:31.268Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.11.0",
"pipeline": "filebeat-7.11.0-postgresql-log-pipeline"
},
"event": {
"module": "postgresql",
"dataset": "postgresql.log"
},
"fileset": {
"name": "log"
},
"deployment": "staging",
"message": "2021-03-16 13:35:12.231 UTC [37] LOG: statement: SELECT \"x\".\"y\", \"x\".\"z\", \"x\".\"expire_date\" FROM \"x\" WHERE (\"x\".\"expire_date\" > '2021-03-16T13:35:12.19
0372+00:00'::timestamptz AND \"x\".\"y\" = 'xyz') LIMIT 21",
"service": {
"type": "postgresql"
}
}
We are using filebeat dockerized, the entry in docker-compose.yml looks like this:
filebeat:
restart: unless-stopped
image: docker.elastic.co/beats/filebeat:7.11.0
user: root
environment:
- DEPLOYMENT_NAME
volumes:
- db_logs:/var/log/postgresql:ro
- ./services/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
- ./services/filebeat/modules.d:/usr/share/filebeat/modules.d:ro
- /var/lib/docker/containers:/var/lib/docker/containers:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
command: >
filebeat -strict.perms=false
-E output.logstash.hosts=["${LOGSTASH_HOST}"]
-E output.logstash.ssl.verification_mode=full
Our filebeat.yml:
filebeat.config:
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
filebeat.autodiscover:
providers:
- type: docker
hints.enabled: true
hints.default_config.enabled: false
processors:
- drop_fields:
fields:
- agent
- docker
- ecs
- host
- input
- log
- stream
- timestamp
ignore_missing: true
- add_fields:
target: ''
fields:
deployment: ${DEPLOYMENT_NAME:n/a}
logging.metrics.enabled: false
monitoring.enabled: false
Just to make sure it's not related to our processors, I tried removing them, but this lead to the same result.
Our modules.d/postgresql.yml:
- module: postgresql
log:
enabled: true
var.paths: ["/var/log/postgresql/postgresql-*.log"]
Since the full filebeat log is too long for this post, I uploaded it here: 2021-03-16T15:02:30.839Z INFO instance/beat.go:660 Home path: [/usr - Pastebin.com