Filebeat 6.6.2 and autodiscover docker log harversting

Elastic Stack Beats
1

On a Windows 10 OS, I have a series of docker containers running java applications. Each container internally logs to:

/usr/app/logs/some_log_file_name.log

A sample log entry would look like:

{"timestamp":"2019-04-02T21:28:16.786Z","level":"INFO","machineName":"api-query","appVersion":"${env:LOGGER_APP_VERSION}","appName":"${env:LOGGER_APP_NAME}","logger":"com.lw.spring.api.common.services.CacheService","message":"Region added: permission-cache","thread":"main"}

I wish to be able to visualize those logs in kibana, but haven't been able to get the right configuration going. I'm using filebeat 6.6.2.

Here's what I have now in my filebeat.yml file:

filebeat.autodiscover:
  providers:
    - type: docker
      host: "tcp://127.0.0.1:2375"
      templates:
        - condition:
            contains:
              docker.container.image: docker-repo
          config:
            - type: docker
              containers:
                ids:
                    - "${data.docker.container.id}"
               
filebeat.inputs:
- type: docker
  enabled: true
  containers:
    ids:
      - "*"
    processors:
      - add_docker_metadata:

When I start the service, there are no ERROR logs on the filebeat log file. Aside from not getting any logs in Kibana I also notice in the logs:

  1. Log entry:
    2019-04-03T15:00:05.486-0400 DEBUG [bus] bus/bus.go:72 filebeat: map[start:true host:172.17.0.2 port:6379 docker:{"container":{"id":"d74054963f2e757e66285725e4e5a8e9ebe65f40a32d72dbd0c0b2e25922f593","image":"redis"

Why would it be picking up the "redis" image when I specified the image should contain "docker-repo"?

  1. Log entry:
    2019-04-03T15:00:05.487-0400 INFO log/input.go:138 Configured paths: [C:\var\lib\docker\containers\d1837b011439a0463b5727ea71317a4e2e674c950321f30acb5511cd305a075c*.log]

What is the meaning of these kind of paths? Are these supposed to be paths on the windows host or inside the docker containers?

Appreciate any help I can get,
Bruno

2

No replies? No direction? Am I way off here? :smiley:

3

@bmmpt, As you have configured both autodiscover provider and docker type input so you are getting each and every container with all images rather than only "docker-repo"

You have configured "docker-repo" under autodiscover provider but in "filebeat.inputs" type "docker" you have mentioned the containers.ids as "*", so "redis" image gets picked.

There is no such importance to use docker autodiscover provider with filebeat.inputs of type docker as you can mention it under "config" parameter of autdiscover provider.

The indentation of "processor" in your config file is not correct as well.

Filebeat docker type input and autodiscover provider consider docker's log under "/var/lib/docker/contianers/" by default. So as you have not mentioned any container's log path so it is taking default path, for this reason your logs are not getting harvested. So you have to configure "path" / "paths" as below

filebeat.autodiscover:
  providers:
    - type: docker
      host: "tcp://127.0.0.1:2375"
      templates:
        - condition:
            contains:
              docker.container.image: docker-repo
          config:
            - type: docker
              containers:
                ids:
                    - "${data.docker.container.id}"
                paths:
                    - /usr/app/logs/*
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.