Error while getting memory usage: error retrieving process stats

Elastic Stack Beats
1

I get the following logs while running Metricbeat 6.2.1 on Docker Swarm. I am hosted on Docker for AWS which uses Moby Linux and that is based on Alpine if that helps. Is this a supported scenario?

2018-02-20T12:12:03.188Z	ERROR	instance/metrics.go:69	Error while getting memory usage: error retrieving process stats
2018-02-20T12:12:03.188Z	ERROR	instance/metrics.go:113	Error retrieving CPU percentages: error retrieving process stats
2018-02-20T12:12:03.188Z	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"info":{"ephemeral_id":"0bd749e7-87ba-4c6f-ab96-4bd459f6b257","uptime":{"ms":1080058}},"memstats":{"gc_next":9093536,"memory_alloc":8263896,"memory_total":424347336}},"libbeat":{"config":{"module":{"running":4}},"pipeline":{"clients":9,"events":{"active":0}}},"system":{"load":{"1":0.17,"15":0.18,"5":0.08,"norm":{"1":0.085,"15":0.09,"5":0.04}}}}}}
2018-02-20T12:12:03.259Z	ERROR	schema/schema.go:41	Error on field 'count': Missing field: count, Error: Key disk_reads not found
2018-02-20T12:12:03.259Z	ERROR	schema/schema.go:41	Error on field 'count': Missing field: count, Error: Key disk_writes not found
2018-02-20T12:12:04.431Z	WARN	elasticsearch/client.go:502	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbe9b2204cf35bc38, ext:1080127366249, loc:(*time.Location)(0x3b6b300)}, Meta:common.MapStr(nil), Fields:common.MapStr{"rabbitmq":common.MapStr{"queue":common.MapStr{"messages":common.MapStr{"total":common.MapStr{"count":104}, "ready":common.MapStr{"count":0}, "unacknowledged":common.MapStr{"count":104}, "persistent":common.MapStr{"count":104}}, "arguments":common.MapStr{}, "state":"running", "auto_delete":false, "vhost":"/", "exclusive":false, "node":"rabbit@53b22045affd", "memory":common.MapStr{"bytes":111928}, "disk":common.MapStr{"reads":common.MapStr{}, "writes":common.MapStr{}}, "durable":true, "consumers":common.MapStr{"count":1, "utilisation":common.MapStr{}}, "name":"employee.attendance.message-builder"}}, "tags":[]string{"Test"}, "docker":common.MapStr{"container":common.MapStr{"image":"rabbitmq:3.7.2-management-alpine@sha256:90038af8f3f971de3bb67f91967a775b843b9c789a998db9524ce097c0bcef1c", "labels":map[string]string{"com.docker.swarm.task":"", "com.docker.swarm.task.id":"yvv0bhf4jv2miwmo3c4i2lubt", "com.docker.swarm.task.name":"bridge-apps-test_rabbitmq.1.yvv0bhf4jv2miwmo3c4i2lubt", "com.docker.stack.namespace":"bridge-apps-test", "com.docker.swarm.node.id":"4cex8urr6fuptr4l9koraqqdz", "com.docker.swarm.service.id":"lc7c5soguneg0ttb2p8ibjvlg", "com.docker.swarm.service.name":"bridge-apps-test_rabbitmq"}, "id":"53b22045affd86307e98bb33fdd82d1452b5ac09b9a0ecdabb498775f4eea35f", "name":"bridge-apps-test_rabbitmq.1.yvv0bhf4jv2miwmo3c4i2lubt"}}, "beat":common.MapStr{"name":"59945cb07e2b", "hostname":"59945cb07e2b", "version":"6.2.1"}, "metricset":common.MapStr{"host":"10.0.1.21:15672", "rtt":3892, "name":"queue", "module":"rabbitmq"}}, Private:interface {}(nil)}, Flags:0x0} (status=400): {"type":"illegal_argument_exception","reason":"Can't merge a non object mapping [docker.container.labels.com.docker.swarm.task] with an object mapping [docker.container.labels.com.docker.swarm.task]"}
2018-02-20T12:12:04.432Z	WARN	elasticsearch/client.go:502	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbe9b2204cf782eba, ext:1080131720937, loc:(*time.Location)(0x3b6b300)}, Meta:common.MapStr(nil), Fields:common.MapStr{"metricset":common.MapStr{"host":"10.0.1.21:15672", "rtt":4458, "name":"node", "module":"rabbitmq"}, "rabbitmq":common.MapStr{"node":common.MapStr{"queue":common.MapStr{"index":common.MapStr{"journal_write":common.MapStr{"count":304}, "read":common.MapStr{"count":0}, "write":common.MapStr{"count":72}}}, "run":common.MapStr{"queue":0}, "type":"disc", "disk":common.MapStr{"free":common.MapStr{"bytes":9871515648, "limit":common.MapStr{"bytes":50000000}}}, "name":"rabbit@53b22045affd", "fd":common.MapStr{"total":1048576, "used":30}, "mnesia":common.MapStr{"disk":common.MapStr{"tx":common.MapStr{"count":21}}, "ram":common.MapStr{"tx":common.MapStr{"count":242}}}, "proc":common.MapStr{"total":1048576, "used":402}, "io":common.MapStr{"file_handle":common.MapStr{"open_attempt":common.MapStr{"avg":common.MapStr{"ms":0}, "count":700}}, "read":common.MapStr{"bytes":1, "count":1, "avg":common.MapStr{"ms":0}}, "reopen":common.MapStr{"count":1}, "seek":common.MapStr{"avg":common.MapStr{"ms":0}, "count":168}, "sync":common.MapStr{"avg":common.MapStr{"ms":2}, "count":218}, "write":common.MapStr{"bytes":218494, "count":218, "avg":common.MapStr{"ms":0}}}, "processors":2, "uptime":949787884, "msg":common.MapStr{"store_read":common.MapStr{"count":0}, "store_write":common.MapStr{"count":0}}, "socket":common.MapStr{"total":943626, "used":3}, "mem":common.MapStr{"limit":common.MapStr{"bytes":1655037952}, "used":common.MapStr{"bytes":94183424}}, "gc":common.MapStr{"num":common.MapStr{"count":8614950}, "reclaimed":common.MapStr{"bytes":111983064400}}}}, "tags":[]string{"Test"}, "docker":common.MapStr{"container":common.MapStr{"id":"53b22045affd86307e98bb33fdd82d1452b5ac09b9a0ecdabb498775f4eea35f", "name":"bridge-apps-test_rabbitmq.1.yvv0bhf4jv2miwmo3c4i2lubt", "image":"rabbitmq:3.7.2-management-alpine@sha256:90038af8f3f971de3bb67f91967a775b843b9c789a998db9524ce097c0bcef1c", "labels":map[string]string{"com.docker.swarm.service.id":"lc7c5soguneg0ttb2p8ibjvlg", "com.docker.swarm.service.name":"bridge-apps-test_rabbitmq", "com.docker.swarm.task":"", "com.docker.swarm.task.id":"yvv0bhf4jv2miwmo3c4i2lubt", "com.docker.swarm.task.name":"bridge-apps-test_rabbitmq.1.yvv0bhf4jv2miwmo3c4i2lubt", "com.docker.stack.namespace":"bridge-apps-test", "com.docker.swarm.node.id":"4cex8urr6fuptr4l9koraqqdz"}}}, "beat":common.MapStr{"name":"59945cb07e2b", "hostname":"59945cb07e2b", "version":"6.2.1"}}, Private:interface {}(nil)}, Flags:0x0} (status=400): {"type":"illegal_argument_exception","reason":"mapper [docker.container.labels.com.docker.swarm.task] of different type, current_type [keyword], merged_type [ObjectMapper]"}
2018-02-20T12:12:33.188Z	ERROR	instance/metrics.go:69	Error while getting memory usage: error retrieving process stats
2018-02-20T12:12:33.188Z	ERROR	instance/metrics.go:113	Error retrieving CPU percentages: error retrieving process stats
1 Like
2

Hello @muhammadrehansaeed,
This should work on all Linux, The log is interested I see a few errors:

  1. The retrieve process stats
  2. The missing count
  3. Can't merge a non object mapping [docker.container.labels.com.docker.swarm.task] with an object mapping [docker.container.labels.com.docker.swarm.task

I have a few questions:

Do you have x-pack monitoring enabled? I think 1, 2, 3 errors might come from the monitoring feature.

3

Also by looking at line 7 of your log, I think you are also experiencing an issue with docker tags see this issue for a workaround.

4

Thanks, the workaround fixes that issue.

5

I'm not using X-Pack monitoring because when your monitoring needs monitoring...

6

There is a problem getting information about the current process, I've created the following issue https://github.com/elastic/beats/issues/6426 to followup.

Also, the current log statement hide the original error, making this problem a bit harder to debug than it should be.

7

Is there a workaround for this second error you have raised an issue for? What is the impact? What module does this affect, system or docker?

8

I believe it should only affect monitoring, so by turning it off it should be fine.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.