Kafka module metricbeat

Elastic Stack Beats
1

Hello,

I'm pretty new to the Elastic stack. I've just setup Elastic 5.1.1 and I'm trying to use the experimental kafka metrics included in metricbeat. I've already bumped up the logging level of the metricbeat to debug.
I keep getting the error: No advertised broker with address :9092 found"

I am certain the kafka broker is listening there. JMX is enabled on the kafka broker.
Can anyone tell me what I could be doing wrong here and where I can find out what's wrong?

Best Regards,
Jo

2

Can you share the module configuration?

For metricbeat kafka module normally you are advised to run one metricbeat instance on each kafka host (still, module can work with remote brokers though). Reasons for installing one metricbeat per kafka broker:

  • kafka API allows to collect partition offsets only for partitions kafka node is leader for, but metricbeat shall mostly collect metrics from localhost only
  • in addition to kafka module you're recommended to use system the module to collect metrics for:
    • CPU/memory/disk io stats/load usage
    • disk usage (to monitor kafka not running out of space)

In kafka a broker can have 2 addresses. The listening address (e.g. used for bootstrapping) and the advertised address. In metricbeat 5.1.1 you will need to configure the advertised address, such that metricbeat can correctly correlate the broker metadata with the broker you're connecting to. Check you kafka server.properties file for listeners and advertised keywords to figure out the hostname to set.

Next metricbeat releases might implement some more sophisticated broker matching support, such that localhost:9200 will hopefully work out of the box for most installations.

3

Here's the metricbeat configuration

###################### Metricbeat Configuration Example #######################

# This file is an example configuration file highlighting only the most common
# options. The metricbeat.full.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/metricbeat/index.html

#==========================  Modules configuration ============================
metricbeat.modules:

#------------------------------- System Module -------------------------------
- module: system
  metricsets:
    # CPU stats
    - cpu

    # System Load stats
    - load

    # Per CPU core stats
    #- core

    # IO stats
    #- diskio

    # Per filesystem stats
    - filesystem

    # File system summary stats
    - fsstat

    # Memory stats
    - memory
    # Network stats
    - network
    # Per process stats
    - process
  enabled: true
 period: 10s
  processes: ['.*']
#------------------------------ ZooKeeper Module -----------------------------
- module: zookeeper
  metricsets: ["mntr"]
  enabled: true
  period: 10s
  hosts: ["localhost:2181"]
#------------------------------ Kafka Module -----------------------------
- module: kafka
  metricsets: ["partition"]
  enabled: true
  period: 10s
  hosts: ["kfkserver:9092"]
#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
name: kafkaserver
# The tags of the shipper are included in their own field with each
# transaction published.
tags:  ["kafka"]

# Optional fields that you can specify to add additional information to the
# output.
fields:
  env: bmt

#================================ Outputs =====================================
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["elasticsearch:9200"]

My metricbeats logs show:

2016-12-09T14:29:53+01:00 DBG  Publish: {
  "@timestamp": "2016-12-09T13:29:53.809Z",
  "beat": {
    "hostname": "kafkaserver",
    "name": "kafkaserver",
    "version": "5.1.1"
  },
  "error": "No advertised broker with address kafkaserver:9092 found",
  "fields": {
    "env": "bmt"
  },
  "kafka": {
    "partition": {}
  },
  "metricset": {
    "host": "kafkaserver:9092",
    "module": "kafka",
    "name": "partition",
    "rtt": 12710
  },
  "tags": [
    "kafka"
  ],
  "type": "metricsets"
}

Thx again,
Jo

4

Hello,

we have " advertised.listeners = PLAINTEXT://kafkaserver:9092 " in the config of the kafka server. We're running Kafka 0.10. Still we're getting the same error code in metricbeat stating:
No advertised broker with address kafkaserver:9092 found
We seem to be getting metricset.rtt with a number back, that's the only one. Is that as expected? Should't we be getting more metrics back?

5

Can you query your brokers with kafkacat:

e.g. in my simple test cluster (shell on kafka0):

$ kafkacat -L -b localhost
Metadata for all topics (from broker -1: kafka0:9092/bootstrap):
 3 brokers:
  broker 2 at kafka2:9092
  broker 1 at kafka1:9092
  broker 0 at kafka0:9092
...

This command shows the actual advertised broker hosts with broker id.

How come your module config uses kfkserver:9092? This a typo?:

- module: kafka
  metricsets: ["partition"]
  enabled: true
  period: 10s
  hosts: ["kfkserver:9092"]

P.S.: please format logs/config files using </> button.

6

Hello Steffen,

I've installed kafkacat and ran it
The kfkserver was indeed a typo, I replaced the actual hostnames for the sake of privacy

kafkacat -L -b kfkolab1.bmt.corp.company.com Metadata for all topics (from broker -1: kfkolab1.bmt.corp.company.com:9092/bootstrap):
3 brokers:
broker 11 at kfkolaA1.bmt.corp.company.com:9092
broker 31 at kfkolaC1.bmt.corp.company.com:9092
broker 21 at kfkolaB1.bmt.corp.company.com:9092
7 topics:
topic "kfkola_dre_raw" with 24 partitions:
...

Best Regards,
Jo

7

So, assuming you're logged into kfkolaA1.bmt.corp.company.com, does ping kfkolaA1.bmt.corp.company.com and telnet kfkolaA1.bmt.corp.company.com 9092 work? If not, name resolution might be a problem.

If ping/telnet do work, this is the address you have to configure in metricbeat running on said host.

8

Hello Steffen,

indeed ping/telnet work as expected.
Still I'm getting warnings in the /var/log/metricbeat/metricbeat

2016-12-13T09:38:34+01:00 INFO Non-zero metrics in the last 30s: fetches.zookeeper-mntr.events=3 libbeat.es.publish.read_bytes=13364 fetches.system-memory.events=3 fetches.system-cpu.events=3 fetches.system-fsstat.events=3 fetches.system-process.success=3 libbeat.es.published_and_acked_events=1249 fetches.system-filesystem.success=3 fetches.system-fsstat.success=3 fetches.system-process.events=1000 fetches.zookeeper-mntr.success=3 fetches.kafka-partition.events=3 fetches.system-load.events=3 libbeat.es.publish.write_bytes=777131 libbeat.publisher.messages_in_worker_queues=1249 libbeat.publisher.published_events=1249 libbeat.es.call_count.PublishEvents=27 fetches.system-cpu.success=3 fetches.system-filesystem.events=222 fetches.system-load.success=3 fetches.system-memory.success=3 fetches.system-network.events=9 fetches.system-network.success=3 fetches.kafka-partition.failures=3
2016-12-13T09:38:34+01:00 WARN Connected to broker at kfkolaa1.bmt.corp.company.com:9092 (unregistered)
2016-12-13T09:38:34+01:00 WARN Closed connection to broker kfkolaa1.bmt.corp.company.com:9092

In ElasticSearch I'm still getting the error:
No advertised broker with address kfkolaa1.bmt.corp.company.com:9092 found

And I only see metricset.name = partition and a metricset.rtt value

Shouldn't I be getting more metrics in ElasticSearch from the kafka module in metricbeat?

We're running metricbeat 5.1.1 and Kafka is at version 0.10.1

Best Regards,
Jo

9

The partitions module tries to match the broker by comparing the host string with the full address string in meta-data (address = <hostname>:<port>).

I can not tell, how much of a typo this is, but you configured kfkolaa1.bmt.corp.company.com:9092, but kafkacat reported kfkolaA1.bmt.corp.company.com:9092.

The string comparison used in code is case-sensitive, which is a bug I think.

10

Thanks Steffen.

By modifying the kafka config to have all lowercase hostnames instead of the uppercase 'A' in there we now get more metrics in ES. We still get the warning the metricbeat log, but I believe that's less important

2016-12-13T14:24:53+01:00 INFO Non-zero metrics in the last 30s: fetches.system-filesystem.success=3 fetches.system-load.events=3 fetches.system-network.success=3 libbeat.es.publish.read_bytes=18480 libbeat.publisher.messages_in_worker_queues=1779 fetches.kafka-partition.success=3 fetches.system-cpu.success=3 fetches.system-memory.success=3 fetches.system-network.events=9 fetches.zookeeper-mntr.success=3 libbeat.es.call_count.PublishEvents=37 libbeat.es.publish.write_bytes=1091865 fetches.system-process.success=3 libbeat.publisher.published_events=1779 fetches.system-cpu.events=3 fetches.zookeeper-mntr.events=3 fetches.system-fsstat.events=3 fetches.system-process.events=1026 fetches.system-load.success=3 libbeat.es.published_and_acked_events=1779 fetches.kafka-partition.events=507 fetches.system-filesystem.events=222 fetches.system-fsstat.success=3 fetches.system-memory.events=3
2016-12-13T14:24:54+01:00 WARN Connected to broker at kfkolaa1.bmt.corp.company.com:9092 (unregistered)

2016-12-13T14:24:54+01:00 WARN Closed connection to broker kfkolaa1.bmt.corp.company.com:9092

Best Regards,
Jo

11

Thanks for reporting in. Will fix the check to be case-insensitive.

Yeah, the warning is by the kafka client library. Unfortunately the library reports all logs without log-level and will always create a warning.

12

Thx Steffen I really appreciate your help

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.