Kafka module metricbeat

Jo_De_Troy · December 9, 2016, 12:33pm

Hello,

I'm pretty new to the Elastic stack. I've just setup Elastic 5.1.1 and I'm trying to use the experimental kafka metrics included in metricbeat. I've already bumped up the logging level of the metricbeat to debug.
I keep getting the error: No advertised broker with address :9092 found"

I am certain the kafka broker is listening there. JMX is enabled on the kafka broker.
Can anyone tell me what I could be doing wrong here and where I can find out what's wrong?

Best Regards,
Jo

steffens · December 9, 2016, 1:20pm

Can you share the module configuration?

For metricbeat kafka module normally you are advised to run one metricbeat instance on each kafka host (still, module can work with remote brokers though). Reasons for installing one metricbeat per kafka broker:

kafka API allows to collect partition offsets only for partitions kafka node is leader for, but metricbeat shall mostly collect metrics from localhost only
in addition to kafka module you're recommended to use system the module to collect metrics for:
- CPU/memory/disk io stats/load usage
- disk usage (to monitor kafka not running out of space)

In kafka a broker can have 2 addresses. The listening address (e.g. used for bootstrapping) and the advertised address. In metricbeat 5.1.1 you will need to configure the advertised address, such that metricbeat can correctly correlate the broker metadata with the broker you're connecting to. Check you kafka server.properties file for listeners and advertised keywords to figure out the hostname to set.

Next metricbeat releases might implement some more sophisticated broker matching support, such that localhost:9200 will hopefully work out of the box for most installations.

Jo_De_Troy · December 9, 2016, 1:32pm

Here's the metricbeat configuration

###################### Metricbeat Configuration Example #######################

# This file is an example configuration file highlighting only the most common
# options. The metricbeat.full.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/metricbeat/index.html

#==========================  Modules configuration ============================
metricbeat.modules:

#------------------------------- System Module -------------------------------
- module: system
  metricsets:
    # CPU stats
    - cpu

    # System Load stats
    - load

    # Per CPU core stats
    #- core

    # IO stats
    #- diskio

    # Per filesystem stats
    - filesystem

    # File system summary stats
    - fsstat

    # Memory stats
    - memory
    # Network stats
    - network
    # Per process stats
    - process
  enabled: true
 period: 10s
  processes: ['.*']
#------------------------------ ZooKeeper Module -----------------------------
- module: zookeeper
  metricsets: ["mntr"]
  enabled: true
  period: 10s
  hosts: ["localhost:2181"]
#------------------------------ Kafka Module -----------------------------
- module: kafka
  metricsets: ["partition"]
  enabled: true
  period: 10s
  hosts: ["kfkserver:9092"]
#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
name: kafkaserver
# The tags of the shipper are included in their own field with each
# transaction published.
tags:  ["kafka"]

# Optional fields that you can specify to add additional information to the
# output.
fields:
  env: bmt

#================================ Outputs =====================================
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["elasticsearch:9200"]

My metricbeats logs show:

2016-12-09T14:29:53+01:00 DBG  Publish: {
  "@timestamp": "2016-12-09T13:29:53.809Z",
  "beat": {
    "hostname": "kafkaserver",
    "name": "kafkaserver",
    "version": "5.1.1"
  },
  "error": "No advertised broker with address kafkaserver:9092 found",
  "fields": {
    "env": "bmt"
  },
  "kafka": {
    "partition": {}
  },
  "metricset": {
    "host": "kafkaserver:9092",
    "module": "kafka",
    "name": "partition",
    "rtt": 12710
  },
  "tags": [
    "kafka"
  ],
  "type": "metricsets"
}

Thx again,
Jo

Jo_De_Troy · December 9, 2016, 3:37pm

Hello,

we have " advertised.listeners = PLAINTEXT://kafkaserver:9092 " in the config of the kafka server. We're running Kafka 0.10. Still we're getting the same error code in metricbeat stating:
No advertised broker with address kafkaserver:9092 found
We seem to be getting metricset.rtt with a number back, that's the only one. Is that as expected? Should't we be getting more metrics back?

steffens · December 9, 2016, 4:46pm

Can you query your brokers with kafkacat:

e.g. in my simple test cluster (shell on kafka0):

$ kafkacat -L -b localhost
Metadata for all topics (from broker -1: kafka0:9092/bootstrap):
 3 brokers:
  broker 2 at kafka2:9092
  broker 1 at kafka1:9092
  broker 0 at kafka0:9092
...

This command shows the actual advertised broker hosts with broker id.

How come your module config uses kfkserver:9092? This a typo?:

- module: kafka
  metricsets: ["partition"]
  enabled: true
  period: 10s
  hosts: ["kfkserver:9092"]

P.S.: please format logs/config files using </> button.

Jo_De_Troy · December 10, 2016, 10:06am

Hello Steffen,

I've installed kafkacat and ran it
The kfkserver was indeed a typo, I replaced the actual hostnames for the sake of privacy

kafkacat -L -b kfkolab1.bmt.corp.company.com Metadata for all topics (from broker -1: kfkolab1.bmt.corp.company.com:9092/bootstrap):
3 brokers:
broker 11 at kfkolaA1.bmt.corp.company.com:9092
broker 31 at kfkolaC1.bmt.corp.company.com:9092
broker 21 at kfkolaB1.bmt.corp.company.com:9092
7 topics:
topic "kfkola_dre_raw" with 24 partitions:
...

Best Regards,
Jo

steffens · December 12, 2016, 1:58pm

So, assuming you're logged into kfkolaA1.bmt.corp.company.com, does ping kfkolaA1.bmt.corp.company.com and telnet kfkolaA1.bmt.corp.company.com 9092 work? If not, name resolution might be a problem.

If ping/telnet do work, this is the address you have to configure in metricbeat running on said host.

Jo_De_Troy · December 13, 2016, 8:56am

Hello Steffen,

indeed ping/telnet work as expected.
Still I'm getting warnings in the /var/log/metricbeat/metricbeat

2016-12-13T09:38:34+01:00 INFO Non-zero metrics in the last 30s: fetches.zookeeper-mntr.events=3 libbeat.es.publish.read_bytes=13364 fetches.system-memory.events=3 fetches.system-cpu.events=3 fetches.system-fsstat.events=3 fetches.system-process.success=3 libbeat.es.published_and_acked_events=1249 fetches.system-filesystem.success=3 fetches.system-fsstat.success=3 fetches.system-process.events=1000 fetches.zookeeper-mntr.success=3 fetches.kafka-partition.events=3 fetches.system-load.events=3 libbeat.es.publish.write_bytes=777131 libbeat.publisher.messages_in_worker_queues=1249 libbeat.publisher.published_events=1249 libbeat.es.call_count.PublishEvents=27 fetches.system-cpu.success=3 fetches.system-filesystem.events=222 fetches.system-load.success=3 fetches.system-memory.success=3 fetches.system-network.events=9 fetches.system-network.success=3 fetches.kafka-partition.failures=3
2016-12-13T09:38:34+01:00 WARN Connected to broker at kfkolaa1.bmt.corp.company.com:9092 (unregistered)
2016-12-13T09:38:34+01:00 WARN Closed connection to broker kfkolaa1.bmt.corp.company.com:9092

In ElasticSearch I'm still getting the error:
No advertised broker with address kfkolaa1.bmt.corp.company.com:9092 found

And I only see metricset.name = partition and a metricset.rtt value

Shouldn't I be getting more metrics in ElasticSearch from the kafka module in metricbeat?

We're running metricbeat 5.1.1 and Kafka is at version 0.10.1

Best Regards,
Jo

steffens · December 13, 2016, 10:59am

The partitions module tries to match the broker by comparing the host string with the full address string in meta-data (address = <hostname>:<port>).

I can not tell, how much of a typo this is, but you configured kfkolaa1.bmt.corp.company.com:9092, but kafkacat reported kfkolaA1.bmt.corp.company.com:9092.

The string comparison used in code is case-sensitive, which is a bug I think.

Jo_De_Troy · December 13, 2016, 1:29pm

Thanks Steffen.

By modifying the kafka config to have all lowercase hostnames instead of the uppercase 'A' in there we now get more metrics in ES. We still get the warning the metricbeat log, but I believe that's less important

2016-12-13T14:24:53+01:00 INFO Non-zero metrics in the last 30s: fetches.system-filesystem.success=3 fetches.system-load.events=3 fetches.system-network.success=3 libbeat.es.publish.read_bytes=18480 libbeat.publisher.messages_in_worker_queues=1779 fetches.kafka-partition.success=3 fetches.system-cpu.success=3 fetches.system-memory.success=3 fetches.system-network.events=9 fetches.zookeeper-mntr.success=3 libbeat.es.call_count.PublishEvents=37 libbeat.es.publish.write_bytes=1091865 fetches.system-process.success=3 libbeat.publisher.published_events=1779 fetches.system-cpu.events=3 fetches.zookeeper-mntr.events=3 fetches.system-fsstat.events=3 fetches.system-process.events=1026 fetches.system-load.success=3 libbeat.es.published_and_acked_events=1779 fetches.kafka-partition.events=507 fetches.system-filesystem.events=222 fetches.system-fsstat.success=3 fetches.system-memory.events=3
2016-12-13T14:24:54+01:00 WARN Connected to broker at kfkolaa1.bmt.corp.company.com:9092 (unregistered)

2016-12-13T14:24:54+01:00 WARN Closed connection to broker kfkolaa1.bmt.corp.company.com:9092

Best Regards,
Jo

steffens · December 13, 2016, 2:41pm

Thanks for reporting in. Will fix the check to be case-insensitive.

Yeah, the warning is by the kafka client library. Unfortunately the library reports all logs without log-level and will always create a warning.

Jo_De_Troy · December 13, 2016, 7:43pm

Thx Steffen I really appreciate your help

system · January 10, 2017, 7:44pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.