Metric beat not running on docker

Gaurav_Agarwal · April 10, 2018, 9:58am

Was trying to install metricbeat on docker to analyse docker containers. But I am getting this error.

2018/04/10 09:51:13.057126 beat.go:297: INFO Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]
2018/04/10 09:51:13.057248 beat.go:192: INFO Setup Beat: metricbeat; Version: 5.6.8
2018/04/10 09:51:13.058260 metrics.go:23: INFO Metrics logging every 30s
2018/04/10 09:51:16.059090 add_cloud_metadata.go:231: INFO add_cloud_metadata: hosting provider type not detected.
2018/04/10 09:51:16.059742 output.go:258: INFO Loading template enabled. Reading template file: /usr/share/metricbeat/metricbeat.template.json
2018/04/10 09:51:16.064707 output.go:269: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /usr/share/metricbeat/metricbeat.template-es2x.json
2018/04/10 09:51:16.069881 output.go:281: INFO Loading template enabled for Elasticsearch 6.x. Reading template file: /usr/share/metricbeat/metricbeat.template-es6x.json
2018/04/10 09:51:16.079627 client.go:128: INFO Elasticsearch url: http://elasticsearch:9200
2018/04/10 09:51:16.079774 outputs.go:108: INFO Activated elasticsearch as output plugin.
2018/04/10 09:51:16.079999 publish.go:300: INFO Publisher name: a4971fa7dfd3
2018/04/10 09:51:16.080335 async.go:63: INFO Flush Interval set to: 1s
2018/04/10 09:51:16.080362 async.go:64: INFO Max Bulk Size set to: 50
2018/04/10 09:51:16.080582 metricbeat.go:28: INFO Register [ModuleFactory:[docker, mongodb, mysql, postgresql, system], MetricSetFactory:[apache/status, ceph/cluster_disk, ceph/cluster_health, ceph/monitor_health, ceph/pool_disk, couchbase/bucket, couchbase/cluster, couchbase/node, docker/container, docker/cpu, docker/diskio, docker/healthcheck, docker/image, docker/info, docker/memory, docker/network, haproxy/info, haproxy/stat, jolokia/jmx, kafka/consumergroup, kafka/partition, mongodb/dbstats, mongodb/status, mysql/status, nginx/stubstatus, php_fpm/pool, postgresql/activity, postgresql/bgwriter, postgresql/database, prometheus/collector, prometheus/stats, redis/info, redis/keyspace, system/core, system/cpu, system/diskio, system/filesystem, system/fsstat, system/load, system/memory, system/network, system/process, system/socket, zookeeper/mntr]]
2018/04/10 09:51:16.082212 socket.go:64: INFO socket process info will only be available for metricbeat because the process is running as a non-root user
2018/04/10 09:51:16.085344 beat.go:233: INFO metricbeat start running.
2018/04/10 09:51:17.089955 tcp.go:26: WARN DNS lookup failure "elasticsearch": lookup elasticsearch: no such host
2018/04/10 09:51:17.090029 single.go:140: ERR Connecting error publishing events (retrying): Get http://elasticsearch:9200: lookup elasticsearch: no such host
2018/04/10 09:51:18.105353 tcp.go:26: WARN DNS lookup failure "elasticsearch": lookup elasticsearch: no such host
2018/04/10 09:51:18.105423 single.go:140: ERR Connecting error publishing events (retrying): Get http://elasticsearch:9200: lookup elasticsearch: no such host
2018/04/10 09:51:20.112554 tcp.go:26: WARN DNS lookup failure "elasticsearch": lookup elasticsearch: no such host
2018/04/10 09:51:20.112626 single.go:140: ERR Connecting error publishing events (retrying): Get http://elasticsearch:9200: lookup elasticsearch: no such host
2018/04/10 09:51:24.127810 tcp.go:26: WARN DNS lookup failure "elasticsearch": lookup elasticsearch: no such host
2018/04/10 09:51:24.127921 single.go:140: ERR Connecting error publishing events (retrying): Get http://elasticsearch:9200: lookup elasticsearch: no such host
2018/04/10 09:51:32.137355 tcp.go:26: WARN DNS lookup failure "elasticsearch": lookup elasticsearch: no such host
2018/04/10 09:51:32.137449 single.go:140: ERR Connecting error publishing events (retrying): Get http://elasticsearch:9200: lookup elasticsearch: no such host
^C2018/04/10 09:51:35.325006 metrics.go:51: INFO Total non-zero values: fetches.system-cpu.events=2 fetches.system-cpu.success=2 fetches.system-filesystem.events=3 fetches.system-filesystem.success=1 fetches.system-fsstat.events=1 fetches.system-fsstat.success=1 fetches.system-load.events=2 fetches.system-load.success=2 fetches.system-memory.events=2 fetches.system-memory.success=2 fetches.system-network.events=4 fetches.system-network.success=2 fetches.system-process.events=2 fetches.system-process.success=2 fetches.system-socket.success=39 libbeat.outputs.messages_dropped=1 libbeat.publisher.messages_in_worker_queues=16 libbeat.publisher.published_events=16
2018/04/10 09:51:35.325051 metrics.go:52: INFO Uptime: 22.276472054s
2018/04/10 09:51:35.325062 beat.go:237: INFO metricbeat stopped.

Please let me know what can be done to eliminate this error.

steffens · April 10, 2018, 11:28am

Please format logs, configs and terminal input/output using the </>-Button or markdown code fences. This forum uses Markdown to format posts. Without proper formatting, it can be very hard to read your posts. Proper formatting helps us to help you.

The lock mentions Metricbeat can not find the Elasticsearch host. That is, there is no network route from Metricbeat to Elasticsearch.

Can you share your configs and details about your docker setup as well?

Is Metricbeat running in a container?

Is Elasticsearch running in a container?

Gaurav_Agarwal · April 10, 2018, 11:48am

yes both metricbeat and elasticsearch are running in containers

steffens · April 10, 2018, 12:35pm

How do you start the containers? With docker, every container has it's own network by default. You need to tell docker, docker-compose or whatever tool you use the metricbeat must be linked with elasticsearch container.

You have custom docker files or do you use some pre-build images?

Gaurav_Agarwal · April 11, 2018, 5:34am

pre-build images

steffens · April 11, 2018, 11:32am

How do you start/run/configure the containers?

Gaurav_Agarwal · April 11, 2018, 12:56pm

yellow open metricbeat-2018.04.11 2_Uek11sSV2mRyyWw8EZnA 5 1 2948 0 1.6mb 1.6mb

It is coming on my elasticsearch indices

steffens · April 11, 2018, 6:24pm

This docker run doesn't seem to be linked to your elasticsearch container. The script fails, because it can not connect to the configured Elasitcsearch host. Either because Elasticsearch is not available, or not reachable.

As you want to run this command only once, also consider to delete the container afterwards. e.g. run with docker run -rm -network container:<id> <command>.

Depending on versions, metricbeat needs to connect to Kibana or Elasticsearch.

Running from yml file? You mean docker-compose.yml?

Gaurav_Agarwal · April 12, 2018, 5:49am

wondering if there is a way where in inside the container I can run the command

Gaurav_Agarwal · April 12, 2018, 9:34am

hello can u tell me how can i add graphs with respect to every container on metricbeats because currently the things which I see on the dashboard are not very clear or say I am not able to interpret what these graphs refer to

Gaurav_Agarwal · April 12, 2018, 9:39am

This is where I have reached now the concern is how to include graphs with respect to containers

steffens · April 12, 2018, 10:44am

Have you checked out the Docker or Container dashboards?

Not all dashboards might fit you out of the box. If the container name/id is known, one can use filters. Filters and all other settings are available in the URL. That is, one can bookmark a dashboard with filters applied.

Visualizations can also be modified/copied and put into new dashboards.

Gaurav_Agarwal · April 12, 2018, 11:21am

So can u help me out how to put dashboards on the basis of container for example suppose i want to check how much cpu utilisation is there for a particular container then how do I do it.

steffens · April 13, 2018, 11:43am

Please format logs, configs and terminal input/output using the </>-Button or markdown code fences. This forum uses Markdown to format posts. Without proper formatting, it can be very hard to read your posts.

When using docker, you want to add the add_docker_metadata processor. This will add container metadata if available. Given your request I would say you only need the process metricset, as that's the only one collecting per process metrics. The other ones will mostly collect host global metrics.

Depending on your setup, you want to add labels to your containers. The add_docker_metadata processor collects container labels as well. You might want to have labels per service type (unless service type is in container name) and labels per 'application' (services/containers supposed to work together). If you have multiple groups setting up containers, also create labels per groups. All these labels can be used for filtering and building per application/group dashboards using filters.

Gaurav_Agarwal · April 13, 2018, 11:58am

I want to monitor my containers for the utilization they take in terms of memory , cpu and time can u please let me know how to do it because the docker module is not working for me

steffens · April 16, 2018, 12:40pm

Which docker module? Did you enable the docker module, or are you trying to the the add_docker_metadata event processor?

You get any errors in your logs?

For docker support, you need to expose the docker socket to metricbeat. Otherwise metricbeat can't query the docker daemon for container meta data.

system · May 14, 2018, 12:54pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.