Missing Docker Meta Data Fields in Kibana


(Steve Richards) #1

Hi,

I am evaluating the 6.0.0-beta2 release of Filebeat as I am interested in the docker metadata feature.

In my filebeat.yml I have added the following in my prospector:

processors:
add_docker_metadata:~

Filebeat is sending the output to a Redis instance which in turn makes it available for Kibana. However, I am not seeing any of the new docker metadata fields in the output.

I am running filebeat in a container and I am mounting both the docker.sock and /var/lib/docker/containers into it - I am seeing logs in Kibana but just without the new fields.

Can someone please help?

Thanks.


(Andrew Kroh) #2

What user is filebeat running as? Most likely it's a permission issue and filebeat cannot read/write to the docker socket. One way to address the issue is to run filebeat as root inside the container (docker run -u root ...).


(Steve Richards) #3

Hi Andrew,

I created a filebeat:6.0.0-beta2 image from a Dockerfile based on the prima/filebeat:5.6.0 one (https://hub.docker.com/r/prima/filebeat/~/dockerfile/) - I updated the Dockerfile to pull the 6.0.0-beta2 Filebeat binary and amended the SHA check etc

Then I created another Dockerfile to reference my filebeat:6.0.0-beta2 image and in that I have the following:

FROM filebeat:6.0.0-beta2
COPY filebeat.yml /filebeat.yml
RUN chmod go-w /filebeat.yml
USER root

If I then attach to the running container I can see that filebeat is being run as root. Like I said, I can see data in Kibana but just not the new metadata fields.

Steve


(Andrew Kroh) #4

Ok, if you were using the official Filebeat container images then by default Filebeat would not be running as root. This is why I made that suggestion before asking for logs.

Can you please check your Filebeat logs and share the config you are using.


(Steve Richards) #5

Here is the config:

#######  Filebeat ####################################
filebeat:
  # List of prospectors to fetch data.
  prospectors:
    -
      type: log
      paths:
        - '/var/lib/docker/containers/*/*.log'
      json.message_key: log
      json.keys_under_root: true
      processors:
        add_docker_metadata: ~
      fields:
         sys: docker
         env: dev
      fields_under_root: true
      ignore_older: 48h
      close_older: 48h
      multiline:
        pattern: '^20|^[a-zA-Z]{3,4}'
        negate: true
        match: after
        max_lines: 5000
        tail_files: false
      force_close_files: true

#registry_file: "/usr/share/filebeat/"

############################# Output ##########################################

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.
output:

  redis:
    # Set the host and port where to find Redis.
    hosts: *********
    port: ****

    # Optional index name. The default is filebeat and generates filebeat keys.
    key: "v3docker"

    # Optional Redis database number where the events are stored
    # The default is 0.
    db: 0

    # Optional Redis database number where the topology is stored
    # The default is 1. It must have a different value than db.
    db_topology: 1

    # Optional password to authenticate with. By default, no
    # password is set.
    password: "***********"

    # Optional Redis initial connection timeout in seconds.
    # The default is 5 seconds.
    timeout: 5

    # Optional interval for reconnecting to failed Redis connections.
    # The default is 1 second.
    reconnect_interval: 1

############################# Logging #########################################

# There are three options for the log ouput: syslog, file, stderr.
# Under Windos systems, the log files are per default sent to the file output,
# under all other system per default to syslog.
logging:

  # Send all logging output to syslog. On Windows default is false, otherwise
  # default is true.
  #to_syslog: true

  # Write all logging output to files. Beats automatically rotate files if rotateeveryb
  # limit is reached.
  to_files: true

  # To enable logging to files, to_files option has to be set to true
  files:
    # The directory where the log files will written to.
    path: "/"

    # The name of the files where the logs are written to.
    name: mybeat

    # Configure log file size limit. If limit is reached, log file will be
    # automatically rotated
    rotateeverybytes: 10485760 # = 10MB

    # Number of rotated log files to keep. Oldest files will be deleted first.
    keepfiles: 5

  # Enable debug output for selected components. To enable all selectors use ["*"]
  # Other available selectors are beat, publish, service
  # Multiple selectors can be chained.
  #selectors: [ ]

  # Sets log level. The default log level is error.
  # Available log levels are: critical, error, warning, info, debug
  level: error

Cannot see any issues in the logs - I also set the level to debug in the config and no problems but should I be looking for something specific?


(Carlos Pérez Aradros) #6

Hi @stevejr, thank you for taking the time to test our beta and report back!

Recently we fixed a bug that may be related to your issue: https://github.com/elastic/beats/pull/5133, will be out soon with filebeat 6.0.0-RC1. Anyway we can do some tests to confirm that's the case, so we discard any other possible issue:

Please run filebeat with this debugging flag: -d 'docker'

  • Could you detail how you start the containers you are watching?

Could you try to run this and see if it get's correctly annotated?: docker run busybox echo hello world

Best regards


(Steve Richards) #7

Hi,

I have tried using the command with -d 'docker' but now I do not see any log messages when I start a new container.

As soon as I stop/restart filebeat without the -d 'docker' flag I see the log messages, still without the metadata.

The containers I am watching are started either as individual containers or as services in a Swarm.

Filebeat is also being started as a service in a Swarm, not sure if that makes a difference?

Thanks,

Steve


(Carlos Pérez Aradros) #8

I don't think it makes a difference,

RC1 is out now, could you test with it?

If that doesn't fix the issue, could you please provide the log output for the first 1-2 minutes?


(Steve Richards) #9

Ok, I have downloaded rc1 and am not not seeing any logs being sent to my Redis output for new containers. Obviously something wrong with my setup but I can't find it.

One question, is the SHA1 file correct for RC1 (Linux 64bit) as when I try to build my Filebeat docker image I am getting a SHA mismatch. The SHA1 file contains value 89cb8a510ecf876d707b803cacef3bc6fd4f536f706d2ef7280d65413a7c9cc65cd0941d2c4eb85a5ba3c01a4718da99bb3c911357a59086819ddb750eb25e2a but when I do a sha1sum on the downloaded tar file I get dbdb7be9a45c47171dc9c06bf55f541a5dbb6d49


(Carlos Pérez Aradros) #10

I think it's a sha512, hashes were changed to it recently, we will have to update link names :slight_smile:

I'm wondering, could you try official images? From your Dockerfile at least I see at least one issue: fields.yml is not in place. In any case that should be in the logs as an error, perhaps you can share the log output for the first 1-2 minutes?


(Bartoszx) #11

Hi Carlos

Same issues here.
Tested with rc1

FROM docker.elastic.co/beats/filebeat:6.0.0-rc1
COPY filebeat.yml /usr/share/filebeat/filebeat.yml

Filebeat logs https://pastebin.com/725VBB4v


(Carlos Pérez Aradros) #12

I'm suspecting Docker version issues at the moment, could you both please share your docker info?


(Bartoszx) #13

Server Version: 17.09.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local nvidia-docker
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: ecwlhsg89qyisazht26w5tcor
Is Manager: false
Node Address:
Manager Addresses:
X:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.10.0-35-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 62.79GiB
Name: gpu2.exam.tld
ID: KIX5:DAQM:D6JW:C7ZV:IG45:DU7I:YCIA:VGAU:HHMC:VHDO:2KJC:IZXZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:

127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support


(Bartoszx) #14

Hi Carlos

Any workaround to this problem?
Do you need any additional info?


(Carlos Pérez Aradros) #15

Hi @bartoszx & @stevejr,

I just noticed that there is something wrong in the pasted config:

      processors:
        add_docker_metadata: ~

should be (notice the -):

      processors:
        - add_docker_metadata: ~

Please let me know if that fixes your issue :slight_smile:


(Bartoszx) #16

Thank you :slight_smile:

Working like a charm


(system) #17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.