Hi.
I configured rabbitmq with option:
management.path_prefix = /rabbit
I would like to collect metrics with rabbitmq module, but it allows me configure endpoint and user/password only. Is it possible to made a new option for path_prefix?
Thank you.
Can you try to specify the path as part of the host? Haven't tried but I think it should work.
Can you share your Metricbeat and rabbitmq config?
Sure.
/etc/rabbitmq/rabbitmq.conf:
## RabbitMQ configuration
## Ref: https://github.com/rabbitmq/rabbitmq-server/blob/master/docs/rabbitmq.conf.example
## Authentification
## Clustering
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
cluster_formation.k8s.address_type = ip
cluster_formation.node_cleanup.interval = 10
cluster_formation.node_cleanup.only_log_warning = false
cluster_partition_handling = autoheal
management.path_prefix = /rabbit
management.load_definitions = /etc/rabbitmq/definitions.json
## The default "guest" user is only permitted to access the server
## via a loopback interface (e.g. localhost)
loopback_users.guest = false
## Memory-based Flow Control threshold
vm_memory_high_watermark.absolute = 256MB
/etc/metricbeat/modules.d/rabbitmq.yaml:
- module: rabbitmq
metricsets: ["node", "queue"]
period: 10s
hosts: ["10.233.53.218:15672/rabbit"]
username: rabbitmetricbeat
password: strongpassThanks for the details. Metricbeat should generate an error event which is sent to ES and potentially also log an error. Could you share both, most importantly the error event sent to ES?
Hi, @ruflin. Here you are:
metricbeat logs:
2018-04-11T14:49:10.689Z INFO instance/beat.go:468 Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]
2018-04-11T14:49:10.690Z INFO instance/beat.go:475 Beat UUID: 6ae14aff-8cb8-4766-8b92-57c71a260be6
2018-04-11T14:49:10.690Z INFO instance/beat.go:213 Setup Beat: metricbeat; Version: 6.2.2
2018-04-11T14:49:13.691Z INFO add_cloud_metadata/add_cloud_metadata.go:297 add_cloud_metadata: hosting provider type not detected.
2018-04-11T14:49:13.691Z INFO elasticsearch/client.go:145 Elasticsearch url: http://elasticsearch:9200
2018-04-11T14:49:13.691Z INFO pipeline/module.go:76 Beat name: metricbeat-677ff5c767-pxjbc
2018-04-11T14:49:13.692Z INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2018-04-11T14:49:13.692Z INFO elasticsearch/client.go:145 Elasticsearch url: http://elasticsearch:9200
2018-04-11T14:49:13.702Z INFO elasticsearch/client.go:690 Connected to Elasticsearch version 6.2.2
2018-04-11T14:49:13.702Z INFO kibana/client.go:69 Kibana url: http://kibana:5601
2018-04-11T14:49:28.432Z INFO instance/beat.go:583 Kibana dashboards successfully loaded.
2018-04-11T14:49:28.432Z INFO instance/beat.go:301 metricbeat start running.
2018-04-11T14:49:28.435Z WARN [cfgwarn] node/node.go:35 EXPERIMENTAL: The rabbitmq node metricset is experimental
2018-04-11T14:49:28.435Z WARN [cfgwarn] queue/queue.go:35 EXPERIMENTAL: The rabbitmq queue metricset is experimental
2018-04-11T14:49:28.435Z INFO cfgfile/reload.go:127 Config reloader started
2018-04-11T14:49:28.437Z WARN [cfgwarn] node/node.go:35 EXPERIMENTAL: The rabbitmq node metricset is experimental
2018-04-11T14:49:28.437Z WARN [cfgwarn] queue/queue.go:35 EXPERIMENTAL: The rabbitmq queue metricset is experimental
2018-04-11T14:49:28.437Z INFO cfgfile/reload.go:258 Starting 3 runners ...
2018-04-11T14:49:28.437Z INFO cfgfile/reload.go:219 Loading of config files completed.
2018-04-11T14:49:29.450Z INFO elasticsearch/client.go:690 Connected to Elasticsearch version 6.2.2
2018-04-11T14:49:29.456Z INFO template/load.go:73 Template already exists and will not be overwritten.
2018-04-11T14:49:43.699Z INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":70,"time":78},"total":{"ticks":360,"time":369,"value":360},"user":{"ticks":290,"time":291}},"info":{"ephemeral_id":"1e21db3a-ca67-4bd3-bbe3-eb5a9a9298b2","uptime":{"ms":33051}},"memstats":{"gc_next":5040384,"memory_alloc":5722760,"memory_total":30976216,"rss":30986240}},"libbeat":{"config":{"module":{"running":3,"starts":3},"reloads":1},"output":{"events":{"acked":375,"batches":8,"total":375},"read":{"bytes":5885},"type":"elasticsearch","write":{"bytes":263927}},"pipeline":{"clients":6,"events":{"active":0,"published":375,"retry":50,"total":375},"queue":{"acked":375}}},"metricbeat":{"kubernetes":{"container":{"events":96,"success":96},"node":{"events":6,"success":6},"pod":{"events":80,"success":80},"state_container":{"events":26,"success":26},"state_deployment":{"events":15,"success":15},"state_node":{"events":3,"success":3},"state_pod":{"events":40,"success":40},"state_replicaset":{"events":17,"success":17},"system":{"events":12,"success":12},"volume":{"events":76,"success":76}},"rabbitmq":{"node":{"events":2,"failures":2},"queue":{"events":2,"failures":2}}},"system":{"cpu":{"cores":4},"load":{"1":1,"15":0.61,"5":0.91,"norm":{"1":0.25,"15":0.1525,"5":0.2275}}}}}}
kibana:
HTTP error 406 in node: 406 Not Acceptable
json:
{
"_index": "metricbeat-6.2.2-2018.04.11",
"_type": "doc",
"_id": "sNkxtWIBaAKsLQ55cu57",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2018-04-11T14:52:18.464Z",
"metricset": {
"name": "node",
"module": "rabbitmq",
"host": "10.233.53.218:15672",
"rtt": 1888
},
"error": {
"message": "HTTP error 406 in node: 406 Not Acceptable"
},
"beat": {
"version": "6.2.2",
"name": "metricbeat-677ff5c767-pxjbc",
"hostname": "metricbeat-677ff5c767-pxjbc"
}
},
"fields": {
"@timestamp": [
"2018-04-11T14:52:18.464Z"
]
},
"highlight": {
"metricset.module": [
"@kibana-highlighted-field@rabbitmq@/kibana-highlighted-field@"
]
},
"sort": [
1523458338464
]
}Thanks, exactly what I was looking for. I'm not too faimiliar with rabbitmq but the 406 seems more like it would come from the queue then the management interface.
I think I now understand the issue. The path overwrites the full path and not only prefixes it. To confirm that, could you try the following?
- module: rabbitmq
metricsets: ["node"]
period: 10s
hosts: ["10.233.53.218:15672/rabbit/api/nodes"]
username: rabbitmetricbeat
password: strongpass
Check if you get node stats with this? If not, can you try to access this url with curl?
If you get the data we will need a feature request on github with the path prefix you suggested initially.
Good day, @ruflin
I'm sorry for such a delay. I've changed config - and now it works! ES now recieving data and i can see it in Kibana.
Should i configure metricset: queue as separate module? If i understand your config correctly only difference will be the URL for queues metrics.
Thank you.
Update:
I've updated config with this:
- module: rabbitmq
metricsets: ["queue"]
period: 10s
hosts: ["10.233.53.218:15672/rabbit/api/queues"]
username: rabbitmetricbeat
password: strongpass
ES successfully recieving metrics, but there is error in metricbeat logs:
2018-04-16T08:27:10.043Z ERROR schema/schema.go:41 Error on field 'count': Missing field: count, Error: Key disk_writes not found
2018-04-16T08:27:10.043Z ERROR schema/schema.go:41 Error on field 'count': Missing field: count, Error: Key disk_reads not foundGlad do hear it works. Yes as a workaround you can configure each as a module for now. But we should also have a config option for the path. Can you open an issue for it on Github? https://github.com/elastic/beats
For the other 2 errors: Please also open a separate issue on Github. Make sure to specify the exact version of Metricbeat and RabbitMQ you use + your RabbitMQ config. It seems the disk_* fields are for whatever reason not available in your endpoint and should be marked as optional on our end.
Done.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.