Setup metricbeat to monitor docker containers

Sauerbrei · November 19, 2020, 12:22pm

Hi there,

I am trying to setup metricbeat inside a local docker environment to write the collected data into an AWS Elastic Search Service. The Service runs and authentication is available via Basic Authentication. Here is my docker setup:

docker-stack.yml

metricbeat:
    image: docker.elastic.co/beats/metricbeat:6.4.0
    volumes:
        - /var/run/docker.sock:/var/run/docker.sock
        - path/to/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml
        - path/to/docker.yml:/usr/share/metricbeat/modules.d/docker.yml
    user: root

metricbeat.yml

metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.dashboards.enabled: true
setup.template.settings:
    index.number_of_shards: 1
    index.codec: best_compression
setup.kibana:
    ssl.enabled: true
    protocol: "https"
    host: "host.on.amazon.com:443"
    path: "/_plugin/kibana"
    username: "yyy"
    password: "xxx"
output.elasticsearch:
    hosts: ["https://host.on.amazon.com:443"]
    username: "yyy"
    password: "xxx"

docker.yml

- module: docker
    metricsets:
        - "container"
        - "cpu"
        - "diskio"
        - "healthcheck"
        - "info"
        - "memory"
        - "network"
    hosts: ["unix:///var/run/docker.sock"]
    period: 10s
    enabled: true

When starting the docker container, I get the following logs:

    docker    | 2020-11-19T12:10:28.317Z INFO    instance/beat.go:544    Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]
    docker    | 2020-11-19T12:10:28.321Z INFO    instance/beat.go:551    Beat UUID: ae7c955a-c679-4c3b-af0f-95c0ba3a9be1
    docker    | 2020-11-19T12:10:28.321Z INFO    [seccomp]       seccomp/seccomp.go:116  Syscall filter successfully installed
    docker    | 2020-11-19T12:10:28.321Z INFO    [beat]  instance/beat.go:768    Beat info       {"system_info": ...}
    docker    | 2020-11-19T12:10:28.321Z INFO    [beat]  instance/beat.go:777    Build info      {"system_info": ...}
    docker    | 2020-11-19T12:10:28.321Z INFO    [beat]  instance/beat.go:780    Go runtime info {"system_info": ...}
    docker    | 2020-11-19T12:10:28.322Z INFO    [beat]  instance/beat.go:784    Host info       {"system_info": ...}
    docker    | 2020-11-19T12:10:28.322Z INFO    [beat]  instance/beat.go:813    Process info    {"system_info": ...}
    docker    | 2020-11-19T12:10:28.322Z INFO    instance/beat.go:273    Setup Beat: metricbeat; Version: 6.4.0
    docker    | 2020-11-19T12:10:28.322Z INFO    elasticsearch/client.go:163     Elasticsearch url: https://host.on.amazon.com:443
    docker    | 2020-11-19T12:10:28.322Z INFO    pipeline/module.go:98   Beat name: f9f8815c3442
    docker    | 2020-11-19T12:10:28.323Z INFO    [monitoring]    log/log.go:114  Starting metrics logging every 30s
    docker    | 2020-11-19T12:10:28.323Z INFO    elasticsearch/client.go:163     Elasticsearch url: https://host.on.amazon.com:443
    docker    | 2020-11-19T12:10:40.612Z INFO    elasticsearch/client.go:708     Connected to Elasticsearch version 7.8.0
    docker    | 2020-11-19T12:10:40.612Z INFO    kibana/client.go:113    Kibana url: https://host.on.amazon.com:443/_plugin/kibana
    docker    | 2020-11-19T12:10:47.202Z INFO    [monitoring]    log/log.go:149  Total non-zero metrics  {"monitoring": ...}
    docker    | 2020-11-19T12:10:47.202Z INFO    [monitoring]    log/log.go:150  Uptime: 18.900561226s
    docker    | 2020-11-19T12:10:47.202Z INFO    [monitoring]    log/log.go:127  Stopping metrics logging.
    docker    | 2020-11-19T12:10:47.203Z INFO    instance/beat.go:358    metricbeat stopped.
    docker    | 2020-11-19T12:10:47.203Z ERROR   instance/beat.go:743    Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/metricbeat/kibana: Failed to import index-pattern: Failed to load directory /usr/share/metricbeat/kibana/6/index-pattern:
    docker    |   error loading /usr/share/metricbeat/kibana/6/index-pattern/metricbeat.json: parsing kibana response: unexpected end of JSON input. Response:
    docker    | 2020-11-19T12:10:09.031Z INFO    instance/beat.go:544    Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]
    docker    | 2020-11-19T12:10:09.037Z INFO    instance/beat.go:551    Beat UUID: 59aeaafb-82f9-4b29-b791-de3276b6eba5
    docker    | 2020-11-19T12:10:09.038Z INFO    [seccomp]       seccomp/seccomp.go:116  Syscall filter successfully installed
    docker    | Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/metricbeat/kibana: Failed to import index-pattern: Failed to load directory /usr/share/metricbeat/kibana/6/index-pattern:
    docker    |   error loading /usr/share/metricbeat/kibana/6/index-pattern/metricbeat.json: parsing kibana response: unexpected end of JSON input. Response:
    docker    | 2020-11-19T12:10:09.038Z INFO    [beat]  instance/beat.go:768    Beat info       {"system_info": ...}
    docker    | 2020-11-19T12:10:09.038Z INFO    [beat]  instance/beat.go:777    Build info      {"system_info": ...}
    docker    | 2020-11-19T12:10:09.038Z INFO    [beat]  instance/beat.go:780    Go runtime info {"system_info": ...}
    docker    | 2020-11-19T12:10:09.038Z INFO    [beat]  instance/beat.go:784    Host info       {"system_info": ...}
    docker    | 2020-11-19T12:10:09.039Z INFO    [beat]  instance/beat.go:813    Process info    {"system_info": ...}
    docker    | 2020-11-19T12:10:09.039Z INFO    instance/beat.go:273    Setup Beat: metricbeat; Version: 6.4.0
    docker    | 2020-11-19T12:10:09.039Z INFO    elasticsearch/client.go:163     Elasticsearch url: https://host.on.amazon.com:443
    docker    | 2020-11-19T12:10:09.039Z INFO    pipeline/module.go:98   Beat name: a6c488013e48
    docker    | 2020-11-19T12:10:09.039Z INFO    [monitoring]    log/log.go:114  Starting metrics logging every 30s
    docker    | 2020-11-19T12:10:09.039Z INFO    elasticsearch/client.go:163     Elasticsearch url: https://host.on.amazon.com:443
    docker    | 2020-11-19T12:10:09.220Z INFO    elasticsearch/client.go:708     Connected to Elasticsearch version 7.8.0
    docker    | 2020-11-19T12:10:09.220Z INFO    kibana/client.go:113    Kibana url: https://host.on.amazon.com:443/_plugin/kibana
    docker    | 2020-11-19T12:10:22.004Z INFO    [monitoring]    log/log.go:149  Total non-zero metrics  {"monitoring": ...}
    docker    | 2020-11-19T12:10:22.004Z INFO    [monitoring]    log/log.go:150  Uptime: 12.990061471s
    docker    | 2020-11-19T12:10:22.004Z INFO    [monitoring]    log/log.go:127  Stopping metrics logging.
    docker    | 2020-11-19T12:10:22.004Z INFO    instance/beat.go:358    metricbeat stopped.
    docker    | 2020-11-19T12:10:22.004Z ERROR   instance/beat.go:743    Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/metricbeat/kibana: Failed to import index-pattern: Failed to load directory /usr/share/metricbeat/kibana/6/index-pattern:
    docker    |   error loading /usr/share/metricbeat/kibana/6/index-pattern/metricbeat.json: parsing kibana response: unexpected end of JSON input. Response:
    docker    | Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/metricbeat/kibana: Failed to import index-pattern: Failed to load directory /usr/share/metricbeat/kibana/6/index-pattern:
    docker    |   error loading /usr/share/metricbeat/kibana/6/index-pattern/metricbeat.json: parsing kibana response: unexpected end of JSON input. Response:

Setting read_only_allow_delete to false did not ne trick. What else can I do?
Thanks for any help!

shaunak · November 23, 2020, 12:29am

Version 6.4.0 is a fairly old release, released over 2 years ago. Is there any reason you are not using a more recent version of Metricbeat, say, Metricbeat 7.10.0?

Sauerbrei · November 23, 2020, 9:21am

yes, there was an error with the license: https://github.com/elastic/beats/issues/16725

system · December 21, 2020, 11:21am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Metricbeat 7.13 not working Beats docker , metricbeat	1	213	October 18, 2023
Metric beat is not pushing data to elastic search - I am using metric beat to monitor host and dockers Beats metricbeat	9	602	November 12, 2019
How to Run Metricbeat in Docker with Modules to Monitor Other Containers Beats elastic-stack-monitoring , docker , beats-module , metricbeat	9	2643	October 2, 2020
Metricbeat configuration not applied correctly Beats docker , metricbeat	2	316	January 20, 2024
How to monitor docker services using metricbeat Beats metricbeat	1	316	June 3, 2019

Setup metricbeat to monitor docker containers

Related topics