Docker filebeat autodiscover not detecting nginx logs

[Fachtna_Simi]

On my mac I am running nginx in a docker file and filebeat in a docker file.

docker run -p 80:80 nginx

The above command successfully runs nginx which I can visit in the browser and the output is printed to the console.

This is my docker file that I am building and then running:

FROM docker.elastic.co/beats/filebeat:7.8.0
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
USER root

And this is my filebeat.yml referenced in the above:

filebeat.autodiscover:
    providers:
      - type: docker
        templates:
          - condition:
              contains:
                docker.container.image: nginx
            config:
              - type: docker
                containers.ids:
                    - "${data.docker.container.id}"
              - module: nginx
                access:
                  enabled: true
                  containers:
                      stream: "stdout"
                error:
                  enabled: true
                  containers:
                      stream: "stderr"
output.console:
    pretty: true  

I run the filebeat docker image using:

sudo docker run -it -v /var/run/docker.sock:/var/run/docker.sock filebeat

The docker.sock stuff is because of an unable to connect to docker socket error I was getting (some kind of docker user error permissions I presume).

All of the above gets the filebeat running inside the docker container.

INFO [autodiscover] autodiscover/autodiscover.go:113 Starting autodiscover manager

However when I visit localhost in the browser and trigger the nginx log, the only output is occasional docker system metrics every 30 seconds:

{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":40,"time":{"ms":43}},"total":{"ticks":100,"time":{"ms":109},"value":100},"user":{"ticks":60,"time":{"ms":66}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":8},"info":{"ephemeral_id":"d3e79d62-6949-4d79-89e8-c595332c18ed","uptime":{"ms":30054}},"memstats":{"gc_next":10249440,"memory_alloc":5520104,"memory_total":17591608,"rss":55390208},"runtime":{"goroutines":23}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"console"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":2},"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}}}}

The nginx log is not being detected. Please help

[shaunak (Shaunak Kashyap)]

Hi @Fachtna_Simi. welcome to the Elastic community forums!

Based on the documentation at https://www.elastic.co/guide/en/beats/filebeat/current/running-on-docker.html, https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover.html#_docker_2, and https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-docker.html the following configuration should work.

Keep your Dockerfile as-is.

Change your filebeat.yml to this:

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.image: nginx
          config:
            - module: nginx
              access:
                enabled: true
                input:
                  type: container
                  paths:
                    - /var/lib/docker/containers/${data.docker.container.id}/*.log
                  stream: stdout
              error:
                enabled: true
                input:
                  type: container
                  paths:
                    - /var/lib/docker/containers/${data.docker.container.id}/*.log
                  stream: stderr

output.console:
  pretty: true

And start up Filebeat like so:

sudo docker run -it -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker/containers/:/var/lib/docker/containers:ro filebeat

Hope that helps,

Shaunak