Metricbeat stops sending metrics when there is no consumer/publisher on a RabbitMQ queue

[wouter.vandenheede]

Hello,

I have an issue with the metricset Queue for RabbitMQ. If there are 1 or more consumers on a queue, there is no problem. Then metrics from this queue comes in Elasticsearch.

As soon as the consumer disappears, no more metric enters the Elasticsearch. Even though there are still messages coming in on this queue.

The ideal would be that every 10 seconds (in my case) an update of all queues is forwarded to Elasticsearch. Such as what is the case with the metricset Exchange.

- module: rabbitmq
  metricsets:
    - node
    - queue
    - exchange
    - connection
  enabled: true
  period: 10s
  hosts: ["localhost:15672"]

This is very annoying because we then have no idea that there is a problem with this queue because messages are filling up because there is no consumer. Then this should be visually in Kibana, which is not possible now.

I use the following versions:

• RabbitMQ 3.7.9
• Erlang 21.1
• Metricbeat 7.1.1

Can anyone help me?

Thanks,
Wouter

[Kaiyan_Sheng]

Hello @wouter.vandenheede, thanks for posting questions here. I tried to reproduce your problem with rabbitmq 3.7.15 and when I first started rabbitmq metricbeat module, without any consumer, I actually got an error in the event from Elasticsearch:

wrong format in consumer_utilisation: expected integer, found

Here is the event from ES:

{
  "_index": "metricbeat-7.1.1-2019.06.14-000001",
  "_type": "_doc",
  "_id": "piRqU2sBClETQ6c-Bdx8",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-06-14T00:36:21.272Z",
    "host": {
      "name": "KaiyanMacBookPro",
      "hostname": "KaiyanMacBookPro",
      "architecture": "x86_64",
      "os": {
        "build": "17G7024",
        "platform": "darwin",
        "version": "10.13.6",
        "family": "darwin",
        "name": "Mac OS X",
        "kernel": "17.7.0"
      },
      "id": "9C7FAB7B-29D1-5926-8E84-158A9CA3E25D"
    },
    "agent": {
      "type": "metricbeat",
      "ephemeral_id": "96125255-6602-4de8-8f8a-b4566013c49d",
      "hostname": "KaiyanMacBookPro",
      "id": "ce47fa8e-9f07-40bb-9b9f-405a69116bdd",
      "version": "7.1.1"
    },
    "ecs": {
      "version": "1.0.0"
    },
    "error": {
      "message": "1 error: 1 error: wrong format in `consumer_utilisation`: expected integer, found <nil>"
    },
    "service": {
      "address": "localhost:15672",
      "type": "rabbitmq"
    },
    "event": {
      "duration": 3164966,
      "dataset": "rabbitmq.queue",
      "module": "rabbitmq"
    },
    "metricset": {
      "name": "queue"
    }
  },
  "fields": {
    "@timestamp": [
      "2019-06-14T00:36:21.272Z"
    ]
  },
  "sort": [
    1560472581272
  ]
}

Do you see any errors in metricbeat log or elasticsearch log or new document/event going into ES but with an error message?

[jsoriano (Jaime Soriano)]

I think this issue is fixed by https://github.com/elastic/beats/pull/12089, that will be released on Metricbeat 7.2.

[Kaiyan_Sheng]

Thanks @jsoriano, I did try with metricbeat from master branch and didn't see the same issue.

[wouter.vandenheede]

Thanks for updating the case.

I also just used the latest version 7.2 and it works!

I did indeed see the same error in my Metricbeat log, but thought it had nothing to do with the problem I had. Because this event in Elasticsearch always arrived at the same time as one of the queues that worked for me. And thought that I would see this event for every queue where it didn't work.

Thank you for finding the root cause.
Wouter