Filebeat on "Docker for AWS" fails to start

Elastic Stack Beats
1

My Elastic stack version is at 6.0.0;
ELK + Filebeat + Metricbeat running in docker swarm mode;
Docker host machines are all ubuntu/xenial64.

I have configured filebeat as described in this blog by Carlos Pérez-Aradros: https://www.elastic.co/blog/enrich-docker-logs-with-filebeat which works fine and I can see data in Kibana.

filebeat.prospectors:
- type: log
  paths:
   - '/var/lib/docker/containers/*/*.log'
  json.message_key: log
  json.keys_under_root: true
  processors:
  - add_docker_metadata: ~
output.elasticsearch:
  hosts: ["elasticsearch:9200"]

Now, has anyone tried getting filebeat working on "docker for AWS (CE)"? '/var/lib/docker/containers' can not be mounted and so filebeat fails to start.

On any node in "Docker for AWS", I get this error:

  • ls: cannot access /var/lib/docker/containers: No such file or directory

for anyone who would like to try, here are my docker compose and filebeat.yml files

I will be grateful if anyone give us a shout if they have got filebeat working on "Docker for AWS"

2

Never tested on AWS but I wonder where AWS CE stores the log files of the containers? Or do the directories exist and it's just an permission problem?

3

It seems Docker for AWS uses it's own logging driver, probably you think you can switch that to use json-file? https://docs.docker.com/engine/admin/logging/json-file/

4

According to this issue on Docker4AWS Github:

when you ssh into a Docker4AWS instance, you are inside a container called shell hosting the SSH server. The shell container does not have the necessary privileges required for mounting a device.

Well, I will need to try --pid option to see if directories from the host can be picked up

5

I have tried --pid option but I still get this error:

"invalid mount config for type "bind": bind source path does not exist"

I will be grateful if anyone can suggest how Filebeat 6.0.0 can be started in swarm mode on Docker for AWS.

6

@exekias @ruflin - This particular issue is resolved and I am pleased to report that Filebeat does work as a global service in docker swarm mode on "Docker for AWS"

There were actually two problems with my configuration that prevented Filebeat from starting:

  1. I had to turn off Auditd Filebeat module in the config file as it turned out that Alpine Linux does not have Auditd package. So, in short, it is not possible to start Auditd module of Filebeat on "Docker for AWS".

  2. To get Filebeat working, I had to set "host" to add_docker_metadata processor

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - '/var/lib/docker/containers/*/*.log'
  json.message_key: log
  json.add_error_key: true
  json.keys_under_root: true
  processors:
    - add_docker_metadata:
        host: "unix:///var/run/docker.sock"

For anyone wishing to take a look at my docker compose and Filebeat config files, here they are on GitHub:

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.