Filebeat autodiscovery for docker seems to miss collecting logs of crashed containers

Elastic Stack Beats
,
1

Hi

We are running a multi-node swarm. If services crashes and produces a log entry with the crash exception, these logs are not forward to our Logstash. Besides, we are able to see these logs with docker log.

Version:
Filebeat: docker.elastic.co/beats/filebeat:6.5.1
Docker: 18.09

Config:

filebeat.yml

logging.metrics.enabled: false

filebeat.registry_file: ${path.data}/registry

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

fields:
  env: ${swarm.environment}

output.logstash:
  hosts: ["${logstash.url}:${logstash.port}"]
  slow_start: true

docker-compose.yml

version: '3.2'

services:
  logstash:
    image: logstash_image
    volumes:
      - /usr/share/logstash/queue/:/usr/share/logstash/queue/
    deploy:
      mode: replicated
      replicas: 1

  filebeat:
    image: logstash_image
    volumes:
     - /var/lib/docker/containers/:/var/lib/docker/containers/:ro
     - /var/run/docker.sock:/var/run/docker.sock:ro
     - /usr/share/filebeat/data/:/usr/share/filebeat/data/
     - /etc/hostname:/etc/hostname:ro
     - /var/log/:/var/log/:ro
    environment:
      swarm.environment: develop
    deploy:
      mode: global

networks:
  default:

Why we are not seeing these logs?

Dear Jan

2

Which modules are you running?

Have you checked filebeat logs for errors?

Have you checked if filebeat is reading the log file (registry file contains offset, log includes info message on Start/Stop of a harvester)?

3

Which modules are you running?

Only system and docker autodiscover

Have you checked filebeat logs for errors?

There is one error which is already report and fixed in the master https://github.com/elastic/beats/pull/9305

Have you checked if filebeat is reading the log file (registry file contains offset, log includes info message on Start/Stop of a harvester)?

I see only logs up to the registry position.

date level path messsage
2018-12-05T14:41:57.938Z INFO log/input.go:138 Configured paths: [/var/lib/docker/containers/02da23a669acb638c061b582999f0a9262e01fce1d2e2624ab745f22c2902b48/*.log]
2018-12-05T14:41:57.938Z INFO input/input.go:114 Starting input of type: docker; ID: 11189854344855006298
2018-12-05T14:41:57.938Z INFO log/harvester.go:254 Harvester started for file: /var/lib/docker/containers/02da23a669acb638c061b582999f0a9262e01fce1d2e2624ab745f22c2902b48/02da23a669acb638c061b582999f0a9262e01fce1d2e2624ab745f22c2902b48-json.log
2018-12-05T14:43:13.375Z INFO input/input.go:149 input ticker stopped
2018-12-05T14:43:13.375Z INFO input/input.go:167 Stopping Input: 11189854344855006298
2018-12-05T14:43:13.375Z INFO log/harvester.go:275 Reader was closed: /var/lib/docker/containers/02da23a669acb638c061b582999f0a9262e01fce1d2e2624ab745f22c2902b48/02da23a669acb638c061b582999f0a9262e01fce1d2e2624ab745f22c2902b48-json.log. Closing.
4

Any progress?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.