Autodiscovery with modules in Docker

Elastic Stack Beats
1

Hi,
I have the deployed ES stack (ES + Kibana + Metricbeat + Filebeat) to a Docker Swarm stack.
That stack contains also some services like MySQL (mariadb), Kafka+Zookeeper, Traefik gateways, etc.
I want to read Docker logs via Filebeat and ingest them in ES.
Since Filebeat has the very nice "modules" feature for the above mentioned tools I wanted to put them on use.
But I didn't managed to do it on Docker.
I map these volumes to Filebeat container:

  • '/var/run/docker.sock:/var/run/docker.sock:rw'
  • '/var/lib/docker/containers:/usr/share/filebeat/source-logs:rw'
    This is my filebeat.yml - the problem is that autodiscovery doesn't enable the modules:
name: ${SHIPPER_NAME:"filebeat"}

logging.level: info
logging.metrics.enabled: true
logging.metrics.period: 60s
logging.to_syslog: false
logging.json: false
  
filebeat.shutdown_timeout: 5

queue.mem:
  events: 8192
  flush.min_events: 0
  flush.timeout: 0s
  
filebeat.config:
  prospectors:
    enabled: true
    path: prospectors.d/*.yml
    reload.enabled: true
    reload.period: 10s
  modules:
    enabled: true
    path: modules.d/*.yml
    reload.enabled: true
    reload.period: 10s


setup.template.enabled: true
setup.template.overwrite: false
setup.template.fields: "fields.yml"
setup.template.name: ${ES_TEMPLATE}
setup.template.pattern: ${ES_TEMPLATE}-*
setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 1

setup.dashboards.enabled: true
setup.dashboards.directory: ${FILEBEAT_HOME}/logs_dashboards
setup.dashboards.beat: filebeat
setup.dashboards.kibana_index: .kibana
setup.dashboards.index: ${ES_TEMPLATE}-*

setup.kibana:
  host: '${KIBANA_URL}'


output.elasticsearch:
  enabled: true
  max_retries: 10
  timeout: 90
  loadbalance: true
  worker: 2
  hosts: '${ELASTICSEARCH_URL}'
  index: '${ES_INDEX}'
  pipeline: '${ES_PIPELINE:""}'


# filebeat.modules:

# - module: mysql
#   error:
#     enabled: true
#   slowlog:
#     enabled: true

# - module: traefik
#   access:
#     enabled: true

# - module: kafka
#   log:
#     enabled: true

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
      - condition:
            contains:
              docker.container.image: "mariadb"
        config:
            - module: mysql
              slowlog:
                enabled: true
                var.paths:
                  - /usr/share/filebeat/source-logs/${data.docker.container.id}/*.log
                input:
                  type: docker
                  containers.ids:
                    - "${data.docker.container.id}"
              error:
                enabled: true
                var.paths:
                  - /usr/share/filebeat/source-logs/${data.docker.container.id}/*.log
                input:
                  type: docker
                  containers.ids:
                    - "${data.docker.container.id}"
      - condition:
            contains:
              docker.container.image: "traefik"
        config:
            - module: traefik
              access:
                enabled: true
                var.paths:
                  - /usr/share/filebeat/source-logs/${data.docker.container.id}/*.log
                input:
                  type: docker
                  containers.ids:
                    - "${data.docker.container.id}"
      - condition:
            contains:
              docker.container.image: "johnnypark/kafka-zookeeper"
        config:
            - module: kafka
              log:
                enabled: true
                var.paths:
                  - /usr/share/filebeat/source-logs/${data.docker.container.id}/*.log
                input:
                  type: docker
                  containers.ids:
                    - "${data.docker.container.id}"

      - condition:
          contains:
            docker.container.image: my_custom_services_image
        config:
          - type: docker
            containers.path: "/usr/share/filebeat/source-logs"
            containers.ids: [ "${data.docker.container.id}" ]
            multiline.pattern: '${MULTILINE_PATTERN:""}'
            multiline.negate: '${MULTILINE_NEGATE_PATTERN:false}'
            multiline.match: '${MULTILINE_MATCH_POSITION:"after"}'

filebeat.prospectors:
  - type: docker
    enabled: true
    containers.path: "/usr/share/filebeat/source-logs"
    containers.ids: [ "*" ]
    multiline.pattern: '${MULTILINE_PATTERN:""}'
    multiline.negate: '${MULTILINE_NEGATE_PATTERN:false}'
    multiline.match: '${MULTILINE_MATCH_POSITION:"after"}'

processors:

- add_locale:
    format: offset

- add_cloud_metadata: ~

- add_docker_metadata:
    host: '${DOCKER_ENDPOINT:unix:///var/run/docker.sock}'
1 Like
2

Looking at the config that should work, do you have any related message in the logs?

3

The logs messages are put in filebeat* index (with metadata fields from docker processor) but without any specific modules processing.
It looks to me that those autodiscover events are not triggered.. the "contains: docker.container.image" maybe is not triggered?
Since you say that the config looks ok, how can I check if those conditions/events works - there are some specific messages in filebeat logs or so?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.