Add docker meta data for docker log files AND harvest other log files

Elastic Stack Beats
1

Hi!
I am evaluating FileBeat 6.0.0-rc1.
It seems as if this configuration:

processors:
- add_docker_metadata:
    host: "unix:///var/run/docker.sock"

works fine and docker fields gets added, but filebeat ignores docker files that are not generated by the docker engine.

Like with this configuration:

  paths:
    - /var/lib/docker/containers/*/*.log # these are harvested fine and docker data is added and sent to output
    - /var/log/*.log   # no data goes to output
    - /var/log/messages # # no data goes to output

Shouldn't it be possible for filebeat to add docker meta data for the docker log files and harvest the other log files without adding the docker meta data, since there is none? I expected that both should be send to output, and only the docker log files should have the docker meta data.

I have tested this both with the config above and by using the system module, with its default config.
System module and the harvester by path /var/lib/docker/containers//.log, both works fine until I config the add_docker_metadata processor, which makes system module stop producing any data to output.

2

Could you share your filebeat log file?

4

Here is some more info and log files

Common to both tests:
harvesting log files from /var/lib/docker/containers/*/*.log
system module enabled so that it collect logs from /var/log/message (among other things)
A docker container logs constantly in reasonable rate to file among /var/lib/docker/containers/*/*.log

Test 1
Setup
Docker meta data collected (enabled)
processors:

- add_docker_metadata:
    host: "unix:///var/run/docker.sock"

Actions
Start filebeat
run command

logger Sending log to /var/log/message

Check trace log file from filebeat for string "Sending log to /var/log/message"
Expected result is that string is there:
Result NOK
Log file:
https://drive.google.com/open?id=0B0lR41uXU8WqVXJWbmpEU0QwQjQ

Test 2
Setup
Docker meta data not collected (disabled)
processors:

#- add_docker_metadata:
#    host: "unix:///var/run/docker.sock"

Actions
Start filebeat
run command

logger Sending log to /var/log/message

Check trace log file from filebeat for string "Sending log to /var/log/message"
Expected result is that string is there:
Result OK

Log file:
https://drive.google.com/open?id=0B0lR41uXU8WqSzlrX2dwM1dyc2M

5

In the logs I see:

drop_fields=docker.container.image, docker.container.labels, prospector, offset, drop_fields=tags, stream, @version, fields}: key=tags: key not found, key=stream: key not found, key=@version: key not found, key=fields: key not found

Did you add a drop_fields processor? It looks like it's causing the issue

6

Yes, I had 2 drop_fields processors in both tests.

processors:
- add_docker_metadata:
    host: "unix:///var/run/docker.sock"
- drop_fields:
    fields: ["docker.container.image","docker.container.labels","prospector","offset"]
- drop_fields:
    fields: ["tags","stream","@version","fields"]
7

I would say that's whats failing here. This is what I would do:

You can define more than one prospector, so you apply the proper processor to each:

filebeat.prospectors:
  - type: log
    paths:
      - /var/lib/docker/containers/*/*.log
    processors:
      - add_docker_metadata: ~
  - type: log
    paths:
      - /var/log/*.log 
      - /var/log/messages
    processors:
      - drop_fields:
        ....
8

OK thanks. Just one more question.
Can I not use drop_fields together with add_docker_metadata at all, or can I use it if it hits fields that actually exists?

9

ah yes, definitely you can, just make sure fields exist when you use that processor

1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.