# Metricbeat and RabbitMQ
**URL:** https://discuss.elastic.co/t/metricbeat-and-rabbitmq/220812
**Category:** Beats
**Tags:** metricbeat
**Created:** [February 25, 2020, 10:01am UTC](https://discuss.elastic.co/t/metricbeat-and-rabbitmq/220812 "2020-02-25T10:01:12Z")
**Posts on this page:** 17
**Page:** 1
### Author:  [@tennaen](https://discuss.elastic.co/u/tennaen)
#### Post date: [February 25, 2020, 10:01am UTC](https://discuss.elastic.co/t/metricbeat-and-rabbitmq/220812/1 "2020-02-25T10:01:13Z")
I've set up rabbitmq monitoring with Metricbeat. And this is what i have in logs:
`error in mapping: error applying schema: 22 errors: key `mnesia\_disk\_tx\_count`not found; key`mnesia\_ram\_tx\_count`not found; key`gc\_bytes\_reclaimed`not found; key`gc\_num`not found; key`io\_file\_handle\_open\_attempt\_avg\_time`not found; key`io\_file\_handle\_open\_attempt\_count`not found; key`io\_read\_avg\_time`not found; key`io\_read\_bytes`not found; key`io\_read\_count`not found; key`io\_read\_count`not found; key`io\_seek\_avg\_time`not found; key`io\_seek\_count`not found; key`io\_sync\_count`not found; key`io\_sync\_avg\_time`not found; key`io\_write\_avg\_time`not found; key`io\_write\_bytes`not found; key`io\_write\_count`not found; key`queue\_index\_write\_count`not found; key`queue\_index\_journal\_write\_count`not found; key`queue\_index\_read\_count`not found; key`msg\_store\_read\_count`not found; key`msg\_store\_write\_count` not found`
RabbitMQ version: 3.3.5
Erlang R16B03-1
---
### Author:  [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)
#### Post date: [February 25, 2020, 11:12am UTC](https://discuss.elastic.co/t/metricbeat-and-rabbitmq/220812/2 "2020-02-25T11:12:12Z")
Hi @tennaen!
It looks like the mapping is Elasticsearch is not updated.
Could you try to delete the mapping from ES and then setup again Metricbeat so as to setup the updated mappings?
Regards,
C.
---
### Author:  [@tennaen](https://discuss.elastic.co/u/tennaen)
#### Post date: [February 25, 2020, 11:51am UTC](https://discuss.elastic.co/t/metricbeat-and-rabbitmq/220812/3 "2020-02-25T11:51:48Z")
@ChrsMark thank you for your response.
I've deleted all entries for metricbeat: indexes and templates - still got same error.
This is my metricbeat.yml elasticoutput configuration:
```
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["ip:9200"]
# Protocol - either `http` (default) or `https`.
#protocol: "https"
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
username: "monitor"
password: "${monitor.password}"
index: "metricbeat-focusdev01"
setup.template.name: "metricbeat-rabbitmq"
setup.template.pattern: "metricbeat-focus*"
```
And rabbitmq.yml configuration:
```
- module: rabbitmq
metricsets:
- exchange
- node
- queue
- connection
period: 10s
hosts: ["10.2.197.85:15672"]
username: focusdev
password: ${focusdev.password}
```
And also i wanted to create custom index for my metricbeat, but default one is created.
---
### Author:  [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)
#### Post date: [February 26, 2020, 8:55am UTC](https://discuss.elastic.co/t/metricbeat-and-rabbitmq/220812/4 "2020-02-26T08:55:21Z")
Maybe the post [Custom index name does not work on metricbeat](https://discuss.elastic.co/t/custom-index-name-does-not-work-on-metricbeat/188977) could be of help.
---
### Author:  [@tennaen](https://discuss.elastic.co/u/tennaen)
#### Post date: [February 26, 2020, 9:44am UTC](https://discuss.elastic.co/t/metricbeat-and-rabbitmq/220812/5 "2020-02-26T09:44:40Z")
But still i have a problem with mappings.
---
### Author:  [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)
#### Post date: [February 26, 2020, 1:51pm UTC](https://discuss.elastic.co/t/metricbeat-and-rabbitmq/220812/6 "2020-02-26T13:51:37Z")
In order to debug it you need to isolate the problem. Did you manage to write events to the defined index?
Could you verify that events are sent to the desired index or not?
If so, is the desired index properly set up? Can you check the mapping and see if the keys that are reported to be failing are in the mapping? (you can do it from Kibana)
---
### Author:  [@tennaen](https://discuss.elastic.co/u/tennaen)
#### Post date: [February 27, 2020, 6:25am UTC](https://discuss.elastic.co/t/metricbeat-and-rabbitmq/220812/7 "2020-02-27T06:25:32Z")
@ChrsMark yes, i've managed to write to defined index.
Index was created by metricbeats using template.
```
{
"rabbitmq" : {
"order" : 0,
"index_patterns" : [
"rabbitmq-*"
],
"settings" : {
"index" : {
"lifecycle" : {
"name" : "rabbitmq-lifecycle-policy",
"rollover_alias" : "rabbitmq-dev"
}
}
},
"mappings" : {
"_routing" : {
"required" : false
},
"numeric_detection" : false,
"dynamic_date_formats" : [
"strict_date_optional_time",
"yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
],
"_meta" : { },
"dynamic" : true,
"_source" : {
"excludes" : [],
"includes" : [],
"enabled" : true
},
"date_detection" : true,
"properties" : { }
},
"aliases" : { }
}
}
```
I cannot find those keys in mappings.
---
### Author:  [@tennaen](https://discuss.elastic.co/u/tennaen)
#### Post date: [February 27, 2020, 8:07am UTC](https://discuss.elastic.co/t/metricbeat-and-rabbitmq/220812/8 "2020-02-27T08:07:27Z")
A little update. When i type GET in dev console
```
GET rabbitmq-dev/_mapping/field/rabbitmq.node.*
```
I get this:
```
{
"rabbitmq-dev--000001" : {
"mappings" : {
"rabbitmq.node.queue.index.write.count" : {
"full_name" : "rabbitmq.node.queue.index.write.count",
"mapping" : {
"count" : {
"type" : "long"
}
}
},
"rabbitmq.node.processors" : {
"full_name" : "rabbitmq.node.processors",
"mapping" : {
"processors" : {
"type" : "long"
}
}
},
"rabbitmq.node.mnesia.disk.tx.count" : {
"full_name" : "rabbitmq.node.mnesia.disk.tx.count",
"mapping" : {
"count" : {
"type" : "long"
}
}
},
"rabbitmq.node.io.write.avg.ms" : {
"full_name" : "rabbitmq.node.io.write.avg.ms",
"mapping" : {
"ms" : {
"type" : "long"
}
}
},
"rabbitmq.node.io.sync.avg.ms" : {
"full_name" : "rabbitmq.node.io.sync.avg.ms",
"mapping" : {
"ms" : {
"type" : "long"
}
}
},
"rabbitmq.node.io.sync.count" : {
"full_name" : "rabbitmq.node.io.sync.count",
"mapping" : {
"count" : {
"type" : "long"
}
}
},
"rabbitmq.node.io.write.count" : {
"full_name" : "rabbitmq.node.io.write.count",
"mapping" : {
"count" : {
"type" : "long"
}
}
},
"rabbitmq.node.msg.store_read.count" : {
"full_name" : "rabbitmq.node.msg.store_read.count",
"mapping" : {
"count" : {
"type" : "long"
}
}
},
"rabbitmq.node.io.read.bytes" : {
"full_name" : "rabbitmq.node.io.read.bytes",
"mapping" : {
"bytes" : {
"type" : "long"
}
}
},
"rabbitmq.node.name" : {
"full_name" : "rabbitmq.node.name",
"mapping" : {
"name" : {
"type" : "keyword",
"ignore_above" : 1024
}
}
},
"rabbitmq.node.io.seek.count" : {
"full_name" : "rabbitmq.node.io.seek.count",
"mapping" : {
"count" : {
"type" : "long"
}
}
},
"rabbitmq.node.proc.used" : {
"full_name" : "rabbitmq.node.proc.used",
"mapping" : {
"used" : {
"type" : "long"
}
}
},
"rabbitmq.node.socket.used" : {
"full_name" : "rabbitmq.node.socket.used",
"mapping" : {
"used" : {
"type" : "long"
}
}
},
"rabbitmq.node.mem.used.bytes" : {
"full_name" : "rabbitmq.node.mem.used.bytes",
"mapping" : {
"bytes" : {
"type" : "long"
}
}
},
"rabbitmq.node.fd.used" : {
"full_name" : "rabbitmq.node.fd.used",
"mapping" : {
"used" : {
"type" : "long"
}
}
},
"rabbitmq.node.mem.limit.bytes" : {
"full_name" : "rabbitmq.node.mem.limit.bytes",
"mapping" : {
"bytes" : {
"type" : "long"
}
}
},
"rabbitmq.node.io.read.count" : {
"full_name" : "rabbitmq.node.io.read.count",
"mapping" : {
"count" : {
"type" : "long"
}
}
},
"rabbitmq.node.gc.reclaimed.bytes" : {
"full_name" : "rabbitmq.node.gc.reclaimed.bytes",
"mapping" : {
"bytes" : {
"type" : "long"
}
}
},
"rabbitmq.node.socket.total" : {
"full_name" : "rabbitmq.node.socket.total",
"mapping" : {
"total" : {
"type" : "long"
}
}
},
"rabbitmq.node.io.write.bytes" : {
"full_name" : "rabbitmq.node.io.write.bytes",
"mapping" : {
"bytes" : {
"type" : "long"
}
}
},
"rabbitmq.node.uptime" : {
"full_name" : "rabbitmq.node.uptime",
"mapping" : {
"uptime" : {
"type" : "long"
}
}
},
"rabbitmq.node.run.queue" : {
"full_name" : "rabbitmq.node.run.queue",
"mapping" : {
"queue" : {
"type" : "long"
}
}
},
"rabbitmq.node.proc.total" : {
"full_name" : "rabbitmq.node.proc.total",
"mapping" : {
"total" : {
"type" : "long"
}
}
},
"rabbitmq.node.io.read.avg.ms" : {
"full_name" : "rabbitmq.node.io.read.avg.ms",
"mapping" : {
"ms" : {
"type" : "long"
}
}
},
"rabbitmq.node.fd.total" : {
"full_name" : "rabbitmq.node.fd.total",
"mapping" : {
"total" : {
"type" : "long"
}
}
},
"rabbitmq.node.msg.store_write.count" : {
"full_name" : "rabbitmq.node.msg.store_write.count",
"mapping" : {
"count" : {
"type" : "long"
}
}
},
"rabbitmq.node.mnesia.ram.tx.count" : {
"full_name" : "rabbitmq.node.mnesia.ram.tx.count",
"mapping" : {
"count" : {
"type" : "long"
}
}
},
"rabbitmq.node.io.seek.avg.ms" : {
"full_name" : "rabbitmq.node.io.seek.avg.ms",
"mapping" : {
"ms" : {
"type" : "long"
}
}
},
"rabbitmq.node.io.reopen.count" : {
"full_name" : "rabbitmq.node.io.reopen.count",
"mapping" : {
"count" : {
"type" : "long"
}
}
},
"rabbitmq.node.type" : {
"full_name" : "rabbitmq.node.type",
"mapping" : {
"type" : {
"type" : "keyword",
"ignore_above" : 1024
}
}
},
"rabbitmq.node.io.file_handle.open_attempt.count" : {
"full_name" : "rabbitmq.node.io.file_handle.open_attempt.count",
"mapping" : {
"count" : {
"type" : "long"
}
}
},
"rabbitmq.node.gc.num.count" : {
"full_name" : "rabbitmq.node.gc.num.count",
"mapping" : {
"count" : {
"type" : "long"
}
}
},
"rabbitmq.node.disk.free.limit.bytes" : {
"full_name" : "rabbitmq.node.disk.free.limit.bytes",
"mapping" : {
"bytes" : {
"type" : "long"
}
}
},
"rabbitmq.node.disk.free.bytes" : {
"full_name" : "rabbitmq.node.disk.free.bytes",
"mapping" : {
"bytes" : {
"type" : "long"
}
}
},
"rabbitmq.node.queue.index.read.count" : {
"full_name" : "rabbitmq.node.queue.index.read.count",
"mapping" : {
"count" : {
"type" : "long"
}
}
},
"rabbitmq.node.io.file_handle.open_attempt.avg.ms" : {
"full_name" : "rabbitmq.node.io.file_handle.open_attempt.avg.ms",
"mapping" : {
"ms" : {
"type" : "long"
}
}
},
"rabbitmq.node.queue.index.journal_write.count" : {
"full_name" : "rabbitmq.node.queue.index.journal_write.count",
"mapping" : {
"count" : {
"type" : "long"
}
}
}
}
}
}
```
And it looks like those keys can be found in mappings.
---
### Author:  [@tennaen](https://discuss.elastic.co/u/tennaen)
#### Post date: [February 27, 2020, 8:11am UTC](https://discuss.elastic.co/t/metricbeat-and-rabbitmq/220812/9 "2020-02-27T08:11:44Z")
For example `mnesia_disk_tx_count` is found, but in format `mnesia.disk.tx.count`
---
### Author:  [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)
#### Post date: [February 27, 2020, 8:33am UTC](https://discuss.elastic.co/t/metricbeat-and-rabbitmq/220812/10 "2020-02-27T08:33:10Z")
I think I found the problem. Could you please provide more logging output? I'm afraid that it hasn't to do with ES mapping (my fault here, sorry) but the error comes at [https://github.com/elastic/beats/blob/b8d10dca8bfe49144fb13b1d71077012464c6e02/metricbeat/module/rabbitmq/node/node.go#L110](https://github.com/elastic/beats/blob/b8d10dca8bfe49144fb13b1d71077012464c6e02/metricbeat/module/rabbitmq/node/node.go#L110). This means that the data Metricbeat gets from Rabbitmq are not able to match the mapping. For instance this had happened with other too at [https://github.com/elastic/beats/pull/6887](https://github.com/elastic/beats/pull/6887). Could you confirm?
C.
---
### Author:  [@tennaen](https://discuss.elastic.co/u/tennaen)
#### Post date: [February 27, 2020, 8:38am UTC](https://discuss.elastic.co/t/metricbeat-and-rabbitmq/220812/11 "2020-02-27T08:38:17Z")
@ChrsMark here is message from metricbeat log:
```
2020-02-27T09:34:15.217+0100 INFO module/wrapper.go:252 Error fetching data for metricset rabbitmq.node: error in mapping: e
rror applying schema: 22 errors: key `mnesia_disk_tx_count` not found; key `mnesia_ram_tx_count` not found; key `msg_store_read_coun
t` not found; key `msg_store_write_count` not found; key `io_read_avg_time` not found; key `io_read_bytes` not found; key `io_read_c
ount` not found; key `io_read_count` not found; key `io_seek_avg_time` not found; key `io_seek_count` not found; key `io_sync_avg_ti
me` not found; key `io_sync_count` not found; key `io_write_avg_time` not found; key `io_write_bytes` not found; key `io_write_count
` not found; key `io_file_handle_open_attempt_count` not found; key `io_file_handle_open_attempt_avg_time` not found; key `gc_bytes_
reclaimed` not found; key `gc_num` not found; key `queue_index_journal_write_count` not found; key `queue_index_read_count` not foun
d; key `queue_index_write_count` not found
2020-02-27T09:34:15.217+0100 DEBUG [processors] processing/processors.go:186 Publish event: {
"@timestamp": "2020-02-27T08:34:15.206Z",
"@metadata": {
"beat": "metricbeat",
"type": "_doc",
"version": "7.6.0"
},
"error": {
"message": "error in mapping: error applying schema: 22 errors: key `mnesia_disk_tx_count` not found; key `mnesia_ram_tx_count`
not found; key `msg_store_read_count` not found; key `msg_store_write_count` not found; key `io_read_avg_time` not found; key `io_re
ad_bytes` not found; key `io_read_count` not found; key `io_read_count` not found; key `io_seek_avg_time` not found; key `io_seek_co
unt` not found; key `io_sync_avg_time` not found; key `io_sync_count` not found; key `io_write_avg_time` not found; key `io_write_by
tes` not found; key `io_write_count` not found; key `io_file_handle_open_attempt_count` not found; key `io_file_handle_open_attempt_
avg_time` not found; key `gc_bytes_reclaimed` not found; key `gc_num` not found; key `queue_index_journal_write_count` not found; ke
y `queue_index_read_count` not found; key `queue_index_write_count` not found"
},
"host": {
"id": "8e36bdfa5ef1438f8057c5ae39f99af4",
"containerized": false,
"name": "osx02656",
"hostname": "osx02656",
"architecture": "x86_64",
"os": {
"family": "redhat",
"name": "Red Hat Enterprise Linux Server",
"kernel": "3.10.0-862.11.6.el7.x86_64",
"codename": "Maipo",
"platform": "rhel",
"version": "7.5 (Maipo)"
}
},
"agent": {
"type": "metricbeat",
"ephemeral_id": "12524069-57ed-4258-abbe-0f23d06a6af3",
"hostname": "osx02656",
"id": "bec5d274-1efd-41f4-bb26-f2c12c40613f",
"version": "7.6.0"
},
"ecs": {
"version": "1.4.0"
},
"event": {
"duration": 11073446,
"dataset": "rabbitmq.node",
"module": "rabbitmq"
},
"metricset": {
"name": "node",
"period": 10000
},
"service": {
"address": "10.2.197.86:15672",
"type": "rabbitmq"
}
}
```
And this one, for queue dataset:
```
2020-02-27T09:34:15.207+0100 DEBUG [schema] schema/schema.go:64 ignoring error for key "pct": wrong format in `consu
mer_utilisation`: expected integer, found string
2020-02-27T09:34:15.207+0100 ERROR [rabbitmq.queue] queue/data.go:97 error in mapping: error applying schema: 2 errors: key `exclusive` not found; key `messages_persistent` not found
2020-02-27T09:34:15.207+0100 DEBUG [schema] schema/schema.go:64 ignoring error for key "pct": wrong format in `consumer_utilisation`: expected integer, found string
2020-02-27T09:34:15.207+0100 ERROR [rabbitmq.queue] queue/data.go:97 error in mapping: error applying schema: 2 errors: key `exclusive` not found; key `messages_persistent` not found
2020-02-27T09:34:15.207+0100 DEBUG [processors] processing/processors.go:186 Publish event: {
"@timestamp": "2020-02-27T08:34:15.199Z",
"@metadata": {
"beat": "metricbeat",
"type": "_doc",
"version": "7.6.0"
},
"host": {
"containerized": false,
"hostname": "osx02656",
"name": "osx02656",
"architecture": "x86_64",
"os": {
"name": "Red Hat Enterprise Linux Server",
"kernel": "3.10.0-862.11.6.el7.x86_64",
"codename": "Maipo",
"platform": "rhel",
"version": "7.5 (Maipo)",
"family": "redhat"
},
"id": "8e36bdfa5ef1438f8057c5ae39f99af4"
},
"agent": {
"version": "7.6.0",
"type": "metricbeat",
"ephemeral_id": "12524069-57ed-4258-abbe-0f23d06a6af3",
"hostname": "osx02656",
"id": "bec5d274-1efd-41f4-bb26-f2c12c40613f"
},
"metricset": {
"name": "queue",
"period": 10000
},
"service": {
"type": "rabbitmq",
"address": "10.2.197.86:15672"
},
"event": {
"dataset": "rabbitmq.queue",
"module": "rabbitmq",
"duration": 7674862
},
"error": {
"message": "error applying schema: 2 errors: key `exclusive` not found; key `messages_persistent` not found"
},
"ecs": {
"version": "1.4.0"
}
}
```
---
### Author:  [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)
#### Post date: [February 27, 2020, 8:43am UTC](https://discuss.elastic.co/t/metricbeat-and-rabbitmq/220812/12 "2020-02-27T08:43:14Z")
Thank you!
It is what I mentioned before. What you get from Rabbitmq does not match the desired mapping. Similar to what was happening at [https://github.com/elastic/beats/pull/6887](https://github.com/elastic/beats/pull/6887).
---
### Author:  [@tennaen](https://discuss.elastic.co/u/tennaen)
#### Post date: [February 27, 2020, 8:44am UTC](https://discuss.elastic.co/t/metricbeat-and-rabbitmq/220812/13 "2020-02-27T08:44:17Z")
So is there anything i can do with it?
---
### Author:  [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)
#### Post date: [February 27, 2020, 8:49am UTC](https://discuss.elastic.co/t/metricbeat-and-rabbitmq/220812/14 "2020-02-27T08:49:32Z")
You can investigate if this is normal on rabbitmq's side. If this is sth that can happen you can open a GH issue/PR suggesting to fix it by making those keys optional, like the other one I posted.
Thanks!
---
### Author:  [@tennaen](https://discuss.elastic.co/u/tennaen)
#### Post date: [February 27, 2020, 10:22am UTC](https://discuss.elastic.co/t/metricbeat-and-rabbitmq/220812/15 "2020-02-27T10:22:02Z")
@ChrsMark looks like this problem is related to RabbitMQ version. I run another metricbeat to monitor RabbitMQ 3.7.14 cluster - no error messages.
---
### Author:  [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)
#### Post date: [February 27, 2020, 12:49pm UTC](https://discuss.elastic.co/t/metricbeat-and-rabbitmq/220812/16 "2020-02-27T12:49:19Z")
Ah yeah, right.
I don't see `3.3.5` listed in the docs :[https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-rabbitmq.html#\_compatibility\_39](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-rabbitmq.html#_compatibility_39)
C.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 26, 2020, 12:49pm UTC](https://discuss.elastic.co/t/metricbeat-and-rabbitmq/220812/17 "2020-03-26T12:49:24Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.