Using AutoDiscover feature for Docker does not work when running in Swarm mode

[darkl0rd]

Running Docker Swarm 18.02 CE + Filebeat 6.2.1.

Filebeat is configured to perform auto-discovery of Docker containers, and ship its logs to logstash. On the logstash side of things however, I see the follow messages for all docker logs:

... "reason"=>"mapper [docker.container.labels.com.docker.swarm.task] of different type, current_type [keyword], merged_type [ObjectMapper]" ...

Looking at the mappings in ElasticSearch:

"docker": {
"properties": {
"container": {
"properties": {
"name": {
"ignore_above": 1024,
"type": "keyword"
},
"labels": {
"type": "object"
},
"image": {
"ignore_above": 1024,
"type": "keyword"
},
"id": {
"ignore_above": 1024,
"type": "keyword"
}
..
..
{
"docker.container.labels": {
"mapping": {
"type": "keyword"
},
"match_mapping_type": "string",
"path_match": "docker.container.labels.*"
}
},

docker swarm labels are in the form of:

"com.docker.swarm.service.name": "my_service_name"

Is this a bug in the provided filebeat mapping?

[exekias (Carlos Pérez Aradros)]

Hi @darkl0rd,

Thank you for testing Filebeat and providing feedback!

It seems mapping failed due to some of your labels, com.docker.swarm.task is currently holding an object, and a single keyword is being indexed (and failing). From what I see you probably have a set of lables like this:

  "com.docker.swarm.task": "",
  "com.docker.swarm.task.id": "xxxxx",
  "com.docker.swarm.task.name": "xxxxx"

This is an issue we have recently fixed (yet to be released), and it's related to the dots in label names and how Elasticsearch stores them.

Something you could do as of today is ignoring the com.docker.swarm.task field from Filebeat side, to avoid mapping errors. You can do that by using the drop_field processor:

processors:
- drop_fields:
  fields:
    - `docker.container.labels.com.docker.swarm.task`

[darkl0rd]

Hi @exekias,

Correct, on the labels; these are the labels as they are added by Docker Swarm.
Good to hear that this was recently fixed, for the record though - this issue can be reproduced on a single swarm node/single filebeat instance. The value of the label 'com.docker.swarm.task' always triggers this errors.

I have implemented your suggested workaround; which as expected resolves the issue.
Minor note for others who might read this, it should be:

processors:
  - drop_fields:
      fields:
        - "docker.containers.labels"

(fields should be indented from drop_fields).

Thanks for your help.