# Add docker meta data for docker log files AND harvest other log files **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [October 12, 2017, 9:34am UTC](https://discuss.elastic.co/t/add-docker-meta-data-for-docker-log-files-and-harvest-other-log-files/103688 "2017-10-12T09:34:29Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Anders\_Gibson](https://avatars.discourse-cdn.com/v4/letter/a/ea5d25/32.png) [@Anders\_Gibson](https://discuss.elastic.co/u/Anders_Gibson)\ **Post date:** [October 12, 2017, 9:34am UTC](https://discuss.elastic.co/t/add-docker-meta-data-for-docker-log-files-and-harvest-other-log-files/103688/1 "2017-10-12T09:34:29Z")
Hi! I am evaluating FileBeat 6.0.0-rc1. It seems as if this configuration: ``` processors: - add_docker_metadata: host: "unix:///var/run/docker.sock" ``` works fine and docker fields gets added, but filebeat ignores docker files that are not generated by the docker engine. Like with this configuration: ``` paths: - /var/lib/docker/containers/*/*.log # these are harvested fine and docker data is added and sent to output - /var/log/*.log # no data goes to output - /var/log/messages # # no data goes to output ``` Shouldn't it be possible for filebeat to add docker meta data for the docker log files and harvest the other log files without adding the docker meta data, since there is none? I expected that both should be send to output, and only the docker log files should have the docker meta data. I have tested this both with the config above and by using the system module, with its default config. System module and the harvester by path /var/lib/docker/containers/_/_.log, both works fine until I config the add\_docker\_metadata processor, which makes system module stop producing any data to output. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [October 13, 2017, 7:58am UTC](https://discuss.elastic.co/t/add-docker-meta-data-for-docker-log-files-and-harvest-other-log-files/103688/2 "2017-10-13T07:58:11Z")
Could you share your filebeat log file? ---
**Author:** ![Anders\_Gibson](https://avatars.discourse-cdn.com/v4/letter/a/ea5d25/32.png) [@Anders\_Gibson](https://discuss.elastic.co/u/Anders_Gibson)\ **Post date:** [October 16, 2017, 12:59pm UTC](https://discuss.elastic.co/t/add-docker-meta-data-for-docker-log-files-and-harvest-other-log-files/103688/4 "2017-10-16T12:59:46Z")
Here is some more info and log files **Common to both tests:** harvesting log files from /var/lib/docker/containers/\*/\*.log system module enabled so that it collect logs from /var/log/message (among other things) A docker container logs constantly in reasonable rate to file among /var/lib/docker/containers/\*/\*.log **Test 1** Setup Docker meta data collected (enabled) processors: ``` - add_docker_metadata: host: "unix:///var/run/docker.sock" ``` Actions Start filebeat run command > logger Sending log to /var/log/message Check trace log file from filebeat for string "Sending log to /var/log/message" Expected result is that string is there: Result NOK Log file: [https://drive.google.com/open?id=0B0lR41uXU8WqVXJWbmpEU0QwQjQ](https://drive.google.com/open?id=0B0lR41uXU8WqVXJWbmpEU0QwQjQ) **Test 2** Setup Docker meta data not collected (disabled) processors: ``` #- add_docker_metadata: # host: "unix:///var/run/docker.sock" ``` Actions Start filebeat run command > logger Sending log to /var/log/message Check trace log file from filebeat for string "Sending log to /var/log/message" Expected result is that string is there: Result OK Log file: [https://drive.google.com/open?id=0B0lR41uXU8WqSzlrX2dwM1dyc2M](https://drive.google.com/open?id=0B0lR41uXU8WqSzlrX2dwM1dyc2M) ---
**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\ **Post date:** [October 16, 2017, 3:41pm UTC](https://discuss.elastic.co/t/add-docker-meta-data-for-docker-log-files-and-harvest-other-log-files/103688/5 "2017-10-16T15:41:24Z")
In the logs I see: `drop_fields=docker.container.image, docker.container.labels, prospector, offset, drop_fields=tags, stream, @version, fields}: key=tags: key not found, key=stream: key not found, key=@version: key not found, key=fields: key not found` Did you add a `drop_fields` processor? It looks like it's causing the issue ---
**Author:** ![Anders\_Gibson](https://avatars.discourse-cdn.com/v4/letter/a/ea5d25/32.png) [@Anders\_Gibson](https://discuss.elastic.co/u/Anders_Gibson)\ **Post date:** [October 18, 2017, 11:18am UTC](https://discuss.elastic.co/t/add-docker-meta-data-for-docker-log-files-and-harvest-other-log-files/103688/6 "2017-10-18T11:18:37Z")
Yes, I had 2 drop\_fields processors in both tests. ``` processors: - add_docker_metadata: host: "unix:///var/run/docker.sock" - drop_fields: fields: ["docker.container.image","docker.container.labels","prospector","offset"] - drop_fields: fields: ["tags","stream","@version","fields"] ``` ---
**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\ **Post date:** [October 18, 2017, 11:34am UTC](https://discuss.elastic.co/t/add-docker-meta-data-for-docker-log-files-and-harvest-other-log-files/103688/7 "2017-10-18T11:34:04Z")
I would say that's whats failing here. This is what I would do: You can define more than one prospector, so you apply the proper processor to each: ```auto filebeat.prospectors: - type: log paths: - /var/lib/docker/containers/*/*.log processors: - add_docker_metadata: ~ - type: log paths: - /var/log/*.log - /var/log/messages processors: - drop_fields: .... ``` ---
**Author:** ![Anders\_Gibson](https://avatars.discourse-cdn.com/v4/letter/a/ea5d25/32.png) [@Anders\_Gibson](https://discuss.elastic.co/u/Anders_Gibson)\ **Post date:** [October 18, 2017, 11:41am UTC](https://discuss.elastic.co/t/add-docker-meta-data-for-docker-log-files-and-harvest-other-log-files/103688/8 "2017-10-18T11:41:06Z")
OK thanks. Just one more question. Can I not use drop\_fields together with add\_docker\_metadata at all, or can I use it if it hits fields that actually exists? ---
**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\ **Post date:** [October 18, 2017, 11:53am UTC](https://discuss.elastic.co/t/add-docker-meta-data-for-docker-log-files-and-harvest-other-log-files/103688/9 "2017-10-18T11:53:58Z")
ah yes, definitely you can, just make sure fields exist when you use that processor ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 15, 2017, 11:54am UTC](https://discuss.elastic.co/t/add-docker-meta-data-for-docker-log-files-and-harvest-other-log-files/103688/10 "2017-11-15T11:54:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.