Filebeat not picking up logs from additional files specified for system module

Elastic Stack Beats
1

TL;DR

We are using the filebeat system module to monitor our Docker logs. It worked correctly until our Docker team moved the log location from /var/log/messages. Now it no longer picks up the logs, even though we've added the new path to the system module. I've tested this on two different hosts, with the same results.

The Long Story...

We are using the filebeat system module to ingest our Docker logs into Elasticsearch
via Logstash. Initially, we our logs were writing to /var/log/messages and all was
right with the world, our Docker logs were making it into ES as expected.

Our Docker team decided to move the logging target for Docker from /var/log/messages to /var/log/docker/docker.log. I added the following to the /etc/filebeat/modules.d/system.yml:

var.paths: ["/var/log/docker/docker.log"]

However, log events are no longer being ingested into ES, as expected. When I
run filebeat in debug mode, I can see the following in the log, but no corresponding
event is appearing in logstash.

2019-08-28T08:19:55.962-0500    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2019-08-28T13:19:50.961Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.3.0",
    "pipeline": "filebeat-7.3.0-system-syslog-pipeline"
  },
  "log": {
    "offset": 13376280,
    "file": {
      "path": "/var/log/docker/docker.log"
    }
  },
  "message": "Aug 28 08:19:43 host001 dockerd-current: time=\"2019-08-28T08:19:43.511217235-05:00\" level=error msg=\"failed to deactivate service binding for container test.1.ymfs6rzvxg8id0hrrapiyc99h\" error=\"network sandbox does not exist for container test.1.ymfs6rzvxg8id0hrrapiyc99h\" module=\"node/agent\"",
  "fileset": {
    "name": "syslog"
  },
  "service": {
    "type": "system"
  },
  "application": {
    "env": "non-prod",
    "name": "docker"
  },
  "ecs": {
    "version": "1.0.1"
  },
  "event": {
    "module": "system",
    "dataset": "system.syslog",
    "timezone": "-05:00"
  },
  "input": {
    "type": "log"
  },
  "host": {
    "name": "host001"
  },
  "agent": {
    "version": "7.3.0",
    "type": "filebeat",
    "ephemeral_id": "d4bd8e1a-4437-46f0-bba1-3e50214e22ef",
    "hostname": "host001",
    "id": "75a8e64e-1e81-4896-8708-da8977ebc2eb"
  }
}

When I run Logstash in debug mode, I'm not seeing any errors that would appear to correspond to this.

I've also attempted to monitor the log directly, rather than through the system module, by adding the following to my filebeat.yml:

filebeat:
  inputs:
    - type: log
      paths:
        - '/var/log/docker/*.log'

However, this also does not appear to load events (I've been deleting /var/lib/filebeat/registry at each restart in order to force a full reload).

Update

Just thought of adding this: Filebeat is running as root, and the log file has the following permissions and ownership:

-rw-r--r-- 1 root root 13425894 Aug 28 09:05 docker.log
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.