Filebeat 7.2.0 Container Input: Error Reading Docker Output

[DougR]

I have installed Filebeat 7.2.0 on a host in our Docker cluster to test the container input.

This is the container input segment of my filebeat.yml for that host:

filebeat:
  inputs:
    - type: container
      paths:
        - /var/lib/docker/containers/*/*.log

However, I am getting the following error in the logs:

Jul 17 11:32:05 <docker-host> filebeat[23063]: 2019-07-17T11:32:05.298-0500        ERROR        log/harvester.go:280        Read line error: invalid CRI log format; File: /var/lib/docker/containers/1adbe9378a731b9b8a96f40b46039476eee8bad9bd410e5b02f3fc2bdf6234d6/container-cached.log
Jul 17 11:32:05 <docker-host> filebeat[23063]: 2019-07-17T11:32:05.303-0500        ERROR        log/harvester.go:280        Read line error: invalid CRI log format; File: /var/lib/docker/containers/34c211f3fd10d6ab445bd65c03bfd22ba87043a67888de64b67a7e03deeef1f7/container-cached.log
Jul 17 11:32:05 <docker-host> filebeat[23063]: 2019-07-17T11:32:05.305-0500        ERROR        log/harvester.go:280        Read line error: invalid CRI log format; File: /var/lib/docker/containers/54af27d74e66a69b4e6fc714d4949aec11bdc9aa584fadd67df69f3a53407cea/container-cached.log
Jul 17 11:32:05 <docker-host> filebeat[23063]: 2019-07-17T11:32:05.313-0500        ERROR        log/harvester.go:280        Read line error: invalid CRI log format; File: /var/lib/docker/containers/8c380bdc6577bd806f662b46ace79ebf692f44dd80c35014c1e81745e3d60075/container-cached.log
Jul 17 11:32:05 <docker-host> filebeat[23063]: 2019-07-17T11:32:05.319-0500        ERROR        log/harvester.go:280        Read line error: invalid CRI log format; File: /var/lib/docker/containers/167f71a9923034dfa435853ee03219b3107ff10d2e6cd2d7a367d05f3e4999fc/container-cached.log

[martinr_ubi (Martin R.)]

@DougR

container-cached.log

Please share a sample of such log files properly formatted or in gist so someone can hopefully assist you.

[DougR]

In doing a further dive, it appears that our logs are being directed to syslogd and some junk characters are being injected into them. We're still using Docker 18.07, which can only send to a single log destination; multi-destination logs weren't introduced until 18.09, I believe. Our Docker admins are checking into this.