Filebeat docker input

Elastic Observability Logs
1

Hello,

I have the following filebeat.yml file:

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

#================================ Logging ======================================
logging.level: debug
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

#----------------------------- Filebeat inputs --------------------------------
filebeat.inputs:
  - type: docker
    containers.ids:
      - "*"
  # - type: log
  #   enabled: true
  #   paths:
  #   - '/var/lib/docker/containers/*/*.log'

#----------------------------- Logstash output --------------------------------
output.logstash:
  hosts: ["logstash:5044"]

There is nothing forwarded to logstash. A previous input that worked fine looked like this:

filebeat.inputs:
    - type: log
      enabled: true
      paths:
      - /usr/share/filebeat/logs/server.log

Nothing seems to work when is use type: docker in the input. The filebeat log looks like this:

2019-07-29T09:24:24.982Z        INFO    log/input.go:138        Configured paths: [/var/lib/docker/containers/*/*.log]
2019-07-29T09:24:24.982Z        INFO    input/input.go:114      Starting input of type: docker; ID: 8760737756938806393
2019-07-29T09:24:24.982Z        DEBUG   [cfgfile]       cfgfile/reload.go:118   Checking module configs from: /usr/share/filebeat/modules.d/*.yml
2019-07-29T09:24:24.983Z        DEBUG   [input] log/input.go:174        Start next scan
2019-07-29T09:24:24.983Z        DEBUG   [cfgfile]       cfgfile/reload.go:132   Number of module configs found: 0
2019-07-29T09:24:24.983Z        INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 1
2019-07-29T09:24:24.983Z        INFO    cfgfile/reload.go:150   Config reloader started
2019-07-29T09:24:24.983Z        DEBUG   [input] log/input.go:195        input states cleaned up. Before: 0, After: 0, Pending: 0
2019-07-29T09:24:24.983Z        DEBUG   [cfgfile]       cfgfile/reload.go:176   Scan for new config files
2019-07-29T09:24:24.983Z        DEBUG   [cfgfile]       cfgfile/reload.go:195   Number of module configs found: 0
2019-07-29T09:24:24.983Z        DEBUG   [reload]        cfgfile/list.go:62      Starting reload procedure, current runners: 0
2019-07-29T09:24:24.983Z        DEBUG   [reload]        cfgfile/list.go:80      Start list: 0, Stop list: 0
2019-07-29T09:24:24.983Z        INFO    cfgfile/reload.go:205   Loading of config files completed.
2019-07-29T09:24:34.983Z        DEBUG   [input] input/input.go:152      Run input
2019-07-29T09:24:34.983Z        DEBUG   [input] log/input.go:174        Start next scan
2019-07-29T09:24:34.983Z        DEBUG   [input] log/input.go:195        input states cleaned up. Before: 0, After: 0, Pending: 0
2

I also tried this:

- type: docker
  combine_partial: true
  containers:
    path: "/var/lib/docker/containers"
    stream: "stdout"
    ids:
      - "*"

Same thing.

3

Hi @mihaijulien,

there's a few things we could check in order to narrow this down:

  • Which filebeat and logstash versions are you using?
  • Is your filebeat process running on the host machine or inside a docker container itself?
  • Can you confirm that /var/lib/docker/containers contains the the container state files on your machine and that the filebeat process has read and list permissions on the directory and its subdirectories?
  • Does running filebeat with the -v option for more verbose logging show anything relevant in the logs?
1 Like
4

Hello @weltenwort

Thank you for your answer, I managed to get the setup up and working. The problem was in my docker-compose file for the filebeat service. The problem was with the permissions.

  filebeat:
    image: docker.elastic.co/beats/filebeat:6.7.1
    container_name: filebeat
    command: filebeat -e -strict.perms=false
    user: root
    ports:
      - 9000:9000
    volumes:
      - ./docker/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml
      - /var/lib/docker/containers/:/var/lib/docker/containers/:ro
      - /var/run/docker.sock:/var/run/docker.sock
    restart: on-failure
    env_file: .env
    depends_on:
      logstash:
        condition: service_healthy

user:root was the solution

5

That's good to hear. :+1: Thank you for sharing your solution.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.