Parse host url error docker event

ncasaux · January 14, 2019, 10:43am

Hello,

I try to use Metricbeat with the autodiscover feature for docker, with following config

Metricbeat.yml:

output.elasticsearch:
  hosts: ["elasticsearch:9200"]

setup.template.name: "metricbeat"
setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 0

metricbeat.modules:
- module: system

metricbeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

Metricbeat is not able to parse docker event. See below what happend when I restarted kibana on docker:

2019-01-14T10:35:24.441Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:204        Got a stop event: map[stop:true host:172.20.0.3 port:5601 docker:{"container":{"id":"003a4b7b7f24a0633a486785c79aa3d1f7e2627f4f55b3694b5bc3d534fa2a89","image":"docker.elastic.co/kibana/kibana:6.5.4","labels":{"co":{"elastic":{"logs/module":"kibana","metrics/module":"kibana"}},"license":"Elastic License","org":{"label-schema":{"build-date":"20181205","license":"GPLv2","name":"kibana","schema-version":"1.0","url":"https://www.elastic.co/products/kibana","vcs-url":"https://github.com/elastic/kibana-docker","vendor":"Elastic","version":"6.5.4"}}},"name":"kibana"}} meta:{"docker":{"container":{"id":"003a4b7b7f24a0633a486785c79aa3d1f7e2627f4f55b3694b5bc3d534fa2a89","image":"docker.elastic.co/kibana/kibana:6.5.4","labels":{"co":{"elastic":{"logs/module":"kibana","metrics/module":"kibana"}},"license":"Elastic License","org":{"label-schema":{"build-date":"20181205","license":"GPLv2","name":"kibana","schema-version":"1.0","url":"https://www.elastic.co/products/kibana","vcs-url":"https://github.com/elastic/kibana-docker","vendor":"Elastic","version":"6.5.4"}}},"name":"kibana"}}} config:[0xc420bcbc80]], generated configs: [0xc420bcbc80]
2019-01-14T10:35:24.441Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:214        Config &{{<nil> } <nil> 0xc4205bf120} is not running
2019-01-14T10:35:24.971Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:162        Got a start event: map[start:true host:172.20.0.3 port:5601 docker:{"container":{"id":"003a4b7b7f24a0633a486785c79aa3d1f7e2627f4f55b3694b5bc3d534fa2a89","image":"docker.elastic.co/kibana/kibana:6.5.4","labels":{"co":{"elastic":{"logs/module":"kibana","metrics/module":"kibana"}},"license":"Elastic License","org":{"label-schema":{"build-date":"20181205","license":"GPLv2","name":"kibana","schema-version":"1.0","url":"https://www.elastic.co/products/kibana","vcs-url":"https://github.com/elastic/kibana-docker","vendor":"Elastic","version":"6.5.4"}}},"name":"kibana"}} meta:{"docker":{"container":{"id":"003a4b7b7f24a0633a486785c79aa3d1f7e2627f4f55b3694b5bc3d534fa2a89","image":"docker.elastic.co/kibana/kibana:6.5.4","labels":{"co":{"elastic":{"logs/module":"kibana","metrics/module":"kibana"}},"license":"Elastic License","org":{"label-schema":{"build-date":"20181205","license":"GPLv2","name":"kibana","schema-version":"1.0","url":"https://www.elastic.co/products/kibana","vcs-url":"https://github.com/elastic/kibana-docker","vendor":"Elastic","version":"6.5.4"}}},"name":"kibana"}}} config:[0xc420b2a750]], generated configs: [0xc420b2a750]
2019-01-14T10:35:24.971Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:235        Got a meta field in the event
2019-01-14T10:35:24.971Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:174        Check failed for config &{{<nil> } <nil> 0xc4205d2ba0}: 1 error: host parsing failed for kibana-status: error parsing URL: empty host, won't start runner

Metricbeat is not able to parse the host, however, the host field is there in the start event.

Metricbeat itself is runnnig on docker as well, with this command:

docker run --name metricbeat \
  -d --rm \
  --network=myBridgeNetwork \
  --mount type=bind,source=/proc,target=/hostfs/proc,readonly \
  --mount type=bind,source=/sys/fs/cgroup,target=/hostfs/sys/fs/cgroup,readonly \
  --mount type=bind,source=/,target=/hostfs,readonly \
  --mount type=bind,source=/var/lib/docker/containers,target=/var/lib/docker/containers,readonly \
  --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock,readonly \
  --mount type=bind,source=/root/metricbeat/metricbeat.yml,target=/usr/share/metricbeat/metricbeat.yml,readonly \
  --user=root \
  docker.elastic.co/beats/metricbeat:6.5.4 -e -d "autodiscover" -system.hostfs=/hostfs

Am I missing something ?

exekias · January 16, 2019, 10:40am

Hi @ncasaux,

Your configuration looks correct, can you please share the labels you are adding to the container you want to monitor? It looks like co.elastic.metrics/hosts is missing, check https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-autodiscover-hints.html#_docker_3 for an example.

Best regards

ncasaux · January 16, 2019, 1:09pm

Hi @exekias,

I changed it to

docker run --name kibana \
  -d --rm \
  -p 5601:5601 \
  --network=myBridgeNetwork \
  --label co.elastic.metrics/module=kibana \
  --label co.elastic.metrics/hosts='localhost:${data.port}' \
  --label co.elastic.logs/module=kibana \
  -e "ELASTICSEARCH_URL=http://elasticsearch:9200" \
  -e "SERVER_BASEPATH=/kibana" \
  docker.elastic.co/kibana/kibana:6.5.4

and there is no more parsing error.

However, as my Kibana is exposed under path /kibana, how can I tell Metricbeat tries to query /kibana/ instead of / ?

exekias · January 17, 2019, 3:15pm

Have you tried by adding the path under the hosts variable? If that doesn't work you would need to rely on raw settings input: https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-autodiscover-hints.html#_literal_co_elastic_logs_raw_literal

ncasaux · January 18, 2019, 3:41pm

Hello @exekias ,

I managed to go a bit further in my analysis, here the current status:

1- When I run Kibana like this (note that the only activated metricset is status):

docker run --name kibana \
  -d --rm \
  -p 5601:5601 \
  --network=myBridgeNetwork \
  --label co.elastic.metrics/module=kibana \
  --label co.elastic.metrics/hosts='${data.host}:${data.port}' \
  --label co.elastic.metrics/metricsets=status \
  --label co.elastic.logs/module=kibana \
  -e "ELASTICSEARCH_URL=http://elasticsearch:9200" \
  -e "SERVER_BASEPATH=/kibana" \
  docker.elastic.co/kibana/kibana:6.5.4

there is no problem:

2019-01-18T15:30:57.373Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:162        Got a start event: map[config:[0xc4207f9e60] start:true host:172.20.0.3 port:5601 docker:{"container":{"id":"0fdbbd78e64d9483159cfb989183da5cb4aea06abd80339cb780306379cb54f0","image":"docker.elastic.co/kibana/kibana:6.5.4","labels":{"co":{"elastic":{"logs/module":"kibana","metrics/hosts":"${data.host}:${data.port}","metrics/metricsets":"status","metrics/module":"kibana"}},"license":"Elastic License","org":{"label-schema":{"build-date":"20181205","license":"GPLv2","name":"kibana","schema-version":"1.0","url":"https://www.elastic.co/products/kibana","vcs-url":"https://github.com/elastic/kibana-docker","vendor":"Elastic","version":"6.5.4"}}},"name":"kibana"}} meta:{"docker":{"container":{"id":"0fdbbd78e64d9483159cfb989183da5cb4aea06abd80339cb780306379cb54f0","image":"docker.elastic.co/kibana/kibana:6.5.4","labels":{"co":{"elastic":{"logs/module":"kibana","metrics/hosts":"${data.host}:${data.port}","metrics/metricsets":"status","metrics/module":"kibana"}},"license":"Elastic License","org":{"label-schema":{"build-date":"20181205","license":"GPLv2","name":"kibana","schema-version":"1.0","url":"https://www.elastic.co/products/kibana","vcs-url":"https://github.com/elastic/kibana-docker","vendor":"Elastic","version":"6.5.4"}}},"name":"kibana"}}}], generated configs: [0xc4207f9e60]
2019-01-18T15:30:57.373Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:235        Got a meta field in the event
2019-01-18T15:30:57.374Z        WARN    [cfgwarn]       status/status.go:53     BETA: The kibana/status metricset is beta
2019-01-18T15:30:57.374Z        DEBUG   [autodiscover]  cfgfile/list.go:62      Starting reload procedure, current runners: 0
2019-01-18T15:30:57.374Z        DEBUG   [autodiscover]  cfgfile/list.go:80      Start list: 1, Stop list: 0
2019-01-18T15:30:57.374Z        WARN    [cfgwarn]       status/status.go:53     BETA: The kibana/status metricset is beta
2019-01-18T15:30:57.374Z        DEBUG   [autodiscover]  cfgfile/list.go:101     Starting runner: kibana [metricsets=1]

2 - When I run Kibana like this (note that the only activated metricset is stats this time, and there is no other difference compared to the previous command):

docker run --name kibana \
  -d --rm \
  -p 5601:5601 \
  --network=myBridgeNetwork \
  --label co.elastic.metrics/module=kibana \
  --label co.elastic.metrics/hosts='${data.host}:${data.port}' \
  --label co.elastic.metrics/metricsets=stats \
  --label co.elastic.logs/module=kibana \
  -e "ELASTICSEARCH_URL=http://elasticsearch:9200" \
  -e "SERVER_BASEPATH=/kibana" \
  docker.elastic.co/kibana/kibana:6.5.4

There is this problem:

2019-01-21T09:44:04.242Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:162        Got a start event: map[config:[0xc420be3d40] start:true host:172.20.0.3 port:5601 docker:{"container":{"id":"dcff0cfd9b7f874908e8a0d362fee82f16d645bf9cf1c6340a338e5e7f13bc0d","image":"docker.elastic.co/kibana/kibana:6.5.4","labels":{"co":{"elastic":{"logs/module":"kibana","metrics/hosts":"${data.host}:${data.port}","metrics/metricsets":"stats","metrics/module":"kibana"}},"license":"Elastic License","org":{"label-schema":{"build-date":"20181205","license":"GPLv2","name":"kibana","schema-version":"1.0","url":"https://www.elastic.co/products/kibana","vcs-url":"https://github.com/elastic/kibana-docker","vendor":"Elastic","version":"6.5.4"}}},"name":"kibana"}} meta:{"docker":{"container":{"id":"dcff0cfd9b7f874908e8a0d362fee82f16d645bf9cf1c6340a338e5e7f13bc0d","image":"docker.elastic.co/kibana/kibana:6.5.4","labels":{"co":{"elastic":{"logs/module":"kibana","metrics/hosts":"${data.host}:${data.port}","metrics/metricsets":"stats","metrics/module":"kibana"}},"license":"Elastic License","org":{"label-schema":{"build-date":"20181205","license":"GPLv2","name":"kibana","schema-version":"1.0","url":"https://www.elastic.co/products/kibana","vcs-url":"https://github.com/elastic/kibana-docker","vendor":"Elastic","version":"6.5.4"}}},"name":"kibana"}}}], generated configs: [0xc420be3d40]
2019-01-21T09:44:04.242Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:235        Got a meta field in the event
2019-01-21T09:44:04.242Z        WARN    [cfgwarn]       stats/stats.go:63       EXPERIMENTAL: The kibana/stats metricset is experimental
2019-01-21T09:44:04.243Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:174        Check failed for config &{{<nil> } <nil> 0xc4205d2420}: 1 error: error making http request: Get http://172.20.0.3:5601/api/status: dial tcp 172.20.0.3:5601: connect: connection refused, won't start runner

It does not make sense to me.
Maybe the error message is not correct ? Maybe a bug in this experimental features ?

Thanks in advance !

system · February 15, 2019, 3:44pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.