# Missing Docker Meta Data Fields in Kibana

**URL:** <https://discuss.elastic.co/t/missing-docker-meta-data-fields-in-kibana/101977>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 27, 2017, 12:26pm UTC](https://discuss.elastic.co/t/missing-docker-meta-data-fields-in-kibana/101977 "2017-09-27T12:26:14Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![stevejr](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stevejr](https://discuss.elastic.co/u/stevejr)\
**Post date:** [September 27, 2017, 12:26pm UTC](https://discuss.elastic.co/t/missing-docker-meta-data-fields-in-kibana/101977/1 "2017-09-27T12:26:14Z")

</div>

Hi,

I am evaluating the 6.0.0-beta2 release of Filebeat as I am interested in the docker metadata feature.

In my filebeat.yml I have added the following in my prospector:

processors:  
add\_docker\_metadata:~

Filebeat is sending the output to a Redis instance which in turn makes it available for Kibana. However, I am not seeing any of the new docker metadata fields in the output.

I am running filebeat in a container and I am mounting both the docker.sock and /var/lib/docker/containers into it - I am seeing logs in Kibana but just without the new fields.

Can someone please help?

Thanks.

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [September 27, 2017, 12:58pm UTC](https://discuss.elastic.co/t/missing-docker-meta-data-fields-in-kibana/101977/2 "2017-09-27T12:58:58Z")

</div>

What user is filebeat running as? Most likely it's a permission issue and filebeat cannot read/write to the docker socket. One way to address the issue is to run filebeat as root inside the container (`docker run -u root ...`).

---

<div class="post-metadata">

**Author:** ![stevejr](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stevejr](https://discuss.elastic.co/u/stevejr)\
**Post date:** [September 27, 2017, 1:07pm UTC](https://discuss.elastic.co/t/missing-docker-meta-data-fields-in-kibana/101977/3 "2017-09-27T13:07:15Z")

</div>

Hi Andrew,

I created a filebeat:6.0.0-beta2 image from a Dockerfile based on the prima/filebeat:5.6.0 one ([https://hub.docker.com/r/prima/filebeat/~/dockerfile/](https://hub.docker.com/r/prima/filebeat/~/dockerfile/)) - I updated the Dockerfile to pull the 6.0.0-beta2 Filebeat binary and amended the SHA check etc

Then I created another Dockerfile to reference my filebeat:6.0.0-beta2 image and in that I have the following:

FROM filebeat:6.0.0-beta2  
COPY filebeat.yml /filebeat.yml  
RUN chmod go-w /filebeat.yml  
USER root

If I then attach to the running container I can see that filebeat is being run as root. Like I said, I can see data in Kibana but just not the new metadata fields.

Steve

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [September 27, 2017, 1:12pm UTC](https://discuss.elastic.co/t/missing-docker-meta-data-fields-in-kibana/101977/4 "2017-09-27T13:12:41Z")

</div>

Ok, if you were using the [official](https://www.elastic.co/guide/en/beats/filebeat/current/running-on-docker.html) Filebeat container images then by default Filebeat would not be running as root. This is why I made that suggestion before asking for logs.

Can you please check your Filebeat logs and share the config you are using.

---

<div class="post-metadata">

**Author:** ![stevejr](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stevejr](https://discuss.elastic.co/u/stevejr)\
**Post date:** [September 27, 2017, 1:42pm UTC](https://discuss.elastic.co/t/missing-docker-meta-data-fields-in-kibana/101977/5 "2017-09-27T13:42:08Z")

</div>

Here is the config:

```
####### Filebeat ####################################
filebeat:
  # List of prospectors to fetch data.
  prospectors:
    -
      type: log
      paths:
        - '/var/lib/docker/containers/*/*.log'
      json.message_key: log
      json.keys_under_root: true
      processors:
        add_docker_metadata: ~
      fields:
         sys: docker
         env: dev
      fields_under_root: true
      ignore_older: 48h
      close_older: 48h
      multiline:
        pattern: '^20|^[a-zA-Z]{3,4}'
        negate: true
        match: after
        max_lines: 5000
        tail_files: false
      force_close_files: true

#registry_file: "/usr/share/filebeat/"

############################# Output ##########################################

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.
output:

  redis:
    # Set the host and port where to find Redis.
    hosts: *********
    port: ****

    # Optional index name. The default is filebeat and generates filebeat keys.
    key: "v3docker"

    # Optional Redis database number where the events are stored
    # The default is 0.
    db: 0

    # Optional Redis database number where the topology is stored
    # The default is 1. It must have a different value than db.
    db_topology: 1

    # Optional password to authenticate with. By default, no
    # password is set.
    password: " ***********"

    # Optional Redis initial connection timeout in seconds.
    # The default is 5 seconds.
    timeout: 5

    # Optional interval for reconnecting to failed Redis connections.
    # The default is 1 second.
    reconnect_interval: 1

############################# Logging #########################################

# There are three options for the log ouput: syslog, file, stderr.
# Under Windos systems, the log files are per default sent to the file output,
# under all other system per default to syslog.
logging:

  # Send all logging output to syslog. On Windows default is false, otherwise
  # default is true.
  #to_syslog: true

  # Write all logging output to files. Beats automatically rotate files if rotateeveryb
  # limit is reached.
  to_files: true

  # To enable logging to files, to_files option has to be set to true
  files:
    # The directory where the log files will written to.
    path: "/"

    # The name of the files where the logs are written to.
    name: mybeat

    # Configure log file size limit. If limit is reached, log file will be
    # automatically rotated
    rotateeverybytes: 10485760 # = 10MB

    # Number of rotated log files to keep. Oldest files will be deleted first.
    keepfiles: 5

  # Enable debug output for selected components. To enable all selectors use ["*"]
  # Other available selectors are beat, publish, service
  # Multiple selectors can be chained.
  #selectors: []

  # Sets log level. The default log level is error.
  # Available log levels are: critical, error, warning, info, debug
  level: error

```

Cannot see any issues in the logs - I also set the level to debug in the config and no problems but should I be looking for something specific?

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [September 27, 2017, 2:15pm UTC](https://discuss.elastic.co/t/missing-docker-meta-data-fields-in-kibana/101977/6 "2017-09-27T14:15:43Z")

</div>

Hi @stevejr, thank you for taking the time to test our beta and report back!

Recently we fixed a bug that may be related to your issue: [https://github.com/elastic/beats/pull/5133](https://github.com/elastic/beats/pull/5133), will be out soon with filebeat 6.0.0-RC1. Anyway we can do some tests to confirm that's the case, so we discard any other possible issue:

Please run `filebeat` with this debugging flag: `-d 'docker'`

- Could you detail how you start the containers you are watching?

Could you try to run this and see if it get's correctly annotated?: `docker run busybox echo hello world`

Best regards

---

<div class="post-metadata">

**Author:** ![stevejr](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stevejr](https://discuss.elastic.co/u/stevejr)\
**Post date:** [September 29, 2017, 11:30am UTC](https://discuss.elastic.co/t/missing-docker-meta-data-fields-in-kibana/101977/7 "2017-09-29T11:30:11Z")

</div>

Hi,

I have tried using the command with -d 'docker' but now I do not see any log messages when I start a new container.

As soon as I stop/restart filebeat without the -d 'docker' flag I see the log messages, still without the metadata.

The containers I am watching are started either as individual containers or as services in a Swarm.

Filebeat is also being started as a service in a Swarm, not sure if that makes a difference?

Thanks,

Steve

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [October 2, 2017, 10:25am UTC](https://discuss.elastic.co/t/missing-docker-meta-data-fields-in-kibana/101977/8 "2017-10-02T10:25:40Z")

</div>

I don't think it makes a difference,

[RC1 is out](https://www.elastic.co/downloads/past-releases/filebeat-6-0-0-rc1) now, could you test with it?

If that doesn't fix the issue, could you please provide the log output for the first 1-2 minutes?

---

<div class="post-metadata">

**Author:** ![stevejr](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stevejr](https://discuss.elastic.co/u/stevejr)\
**Post date:** [October 2, 2017, 5:21pm UTC](https://discuss.elastic.co/t/missing-docker-meta-data-fields-in-kibana/101977/9 "2017-10-02T17:21:19Z")

</div>

Ok, I have downloaded rc1 and am not not seeing any logs being sent to my Redis output for new containers. Obviously something wrong with my setup but I can't find it.

One question, is the SHA1 file correct for RC1 (Linux 64bit) as when I try to build my Filebeat docker image I am getting a SHA mismatch. The SHA1 file contains value **89cb8a510ecf876d707b803cacef3bc6fd4f536f706d2ef7280d65413a7c9cc65cd0941d2c4eb85a5ba3c01a4718da99bb3c911357a59086819ddb750eb25e2a** but when I do a sha1sum on the downloaded tar file I get **dbdb7be9a45c47171dc9c06bf55f541a5dbb6d49**

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [October 3, 2017, 8:34am UTC](https://discuss.elastic.co/t/missing-docker-meta-data-fields-in-kibana/101977/10 "2017-10-03T08:34:36Z")

</div>

I think it's a sha512, hashes were changed to it recently, we will have to update link names 🙂

I'm wondering, could you try official images? From your Dockerfile at least I see at least one issue: fields.yml is not in place. In any case that should be in the logs as an error, perhaps you can share the log output for the first 1-2 minutes?

---

<div class="post-metadata">

**Author:** ![bartoszx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bartoszx/32/22971_2.png) [@bartoszx](https://discuss.elastic.co/u/bartoszx)\
**Post date:** [October 11, 2017, 6:30pm UTC](https://discuss.elastic.co/t/missing-docker-meta-data-fields-in-kibana/101977/11 "2017-10-11T18:30:24Z")

</div>

Hi Carlos

Same issues here.  
Tested with rc1

FROM [docker.elastic.co/beats/filebeat:6.0.0-rc1](http://docker.elastic.co/beats/filebeat:6.0.0-rc1)  
COPY filebeat.yml /usr/share/filebeat/filebeat.yml

Filebeat logs [https://pastebin.com/725VBB4v](https://pastebin.com/725VBB4v)

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [October 12, 2017, 2:27am UTC](https://discuss.elastic.co/t/missing-docker-meta-data-fields-in-kibana/101977/12 "2017-10-12T02:27:14Z")

</div>

I'm suspecting Docker version issues at the moment, could you both please share your `docker info`?

---

<div class="post-metadata">

**Author:** ![bartoszx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bartoszx/32/22971_2.png) [@bartoszx](https://discuss.elastic.co/u/bartoszx)\
**Post date:** [October 12, 2017, 4:53am UTC](https://discuss.elastic.co/t/missing-docker-meta-data-fields-in-kibana/101977/13 "2017-10-12T04:53:57Z")

</div>

Server Version: 17.09.0-ce  
Storage Driver: overlay2  
Backing Filesystem: extfs  
Supports d\_type: true  
Native Overlay Diff: true  
Logging Driver: json-file  
Cgroup Driver: cgroupfs  
Plugins:  
Volume: local nvidia-docker  
Network: bridge host macvlan null overlay  
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog  
Swarm: active  
NodeID: ecwlhsg89qyisazht26w5tcor  
Is Manager: false  
Node Address:  
Manager Addresses:  
X:2377  
Runtimes: runc  
Default Runtime: runc  
Init Binary: docker-init  
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0  
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64  
init version: 949e6fa  
Security Options:  
apparmor  
seccomp  
Profile: default  
Kernel Version: 4.10.0-35-generic  
Operating System: Ubuntu 16.04.3 LTS  
OSType: linux  
Architecture: x86\_64  
CPUs: 8  
Total Memory: 62.79GiB  
Name: gpu2.exam.tld  
ID: KIX5:DAQM:D6JW:C7ZV:IG45:DU7I:YCIA:VGAU:HHMC:VHDO:2KJC:IZXZ  
Docker Root Dir: /var/lib/docker  
Debug Mode (client): false  
Debug Mode (server): false  
Registry: [https://index.docker.io/v1/](https://index.docker.io/v1/)  
Experimental: false  
Insecure Registries:

127.0.0.0/8  
Live Restore Enabled: false

WARNING: No swap limit support

---

<div class="post-metadata">

**Author:** ![bartoszx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bartoszx/32/22971_2.png) [@bartoszx](https://discuss.elastic.co/u/bartoszx)\
**Post date:** [October 22, 2017, 4:02pm UTC](https://discuss.elastic.co/t/missing-docker-meta-data-fields-in-kibana/101977/14 "2017-10-22T16:02:46Z")

</div>

Hi Carlos

Any workaround to this problem?  
Do you need any additional info?

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [October 22, 2017, 6:09pm UTC](https://discuss.elastic.co/t/missing-docker-meta-data-fields-in-kibana/101977/15 "2017-10-22T18:09:14Z")

</div>

Hi @bartoszx & @stevejr,

I just noticed that there is something wrong in the pasted config:

```auto
      processors:
        add_docker_metadata: ~

```

should be (notice the `-`):

```auto
      processors:
        - add_docker_metadata: ~

```

Please let me know if that fixes your issue 🙂

---

<div class="post-metadata">

**Author:** ![bartoszx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bartoszx/32/22971_2.png) [@bartoszx](https://discuss.elastic.co/u/bartoszx)\
**Post date:** [October 22, 2017, 7:12pm UTC](https://discuss.elastic.co/t/missing-docker-meta-data-fields-in-kibana/101977/16 "2017-10-22T19:12:24Z")

</div>

Thank you 🙂

Working like a charm

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 19, 2017, 7:12pm UTC](https://discuss.elastic.co/t/missing-docker-meta-data-fields-in-kibana/101977/17 "2017-11-19T19:12:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.