Kafka module error

Elastic Stack Beats
,
1

Hi all,

I try to use the producer, consumer and broker metricset for Kafka. But when I use one of them I have this error:

2020-05-04T11:06:25.118Z	INFO	instance/beat.go:445	metricbeat stopped.
2020-05-04T11:06:25.118Z	ERROR	instance/beat.go:933	Exiting: 1 error: metricset 'kafka/broker' not found
Exiting: 1 error: metricset 'kafka/broker' not found

And my config:

- module: kafka
  metricsets:
    - broker
  period: 10s
  hosts: ["localhost:8779"]

It's as if this module doesn't exist in v7.6.0 of metricbeat but the doc says the opposite. Any ideas ?

I'm using version 7.6.0 of metricbeat and version 1.17 of Kubernetes.

2

from the logs, it looks like the metric can't find the broker.
is the port 8779 listening on the server?

3

Hi,

I'm using this, but I have the same error:

- name: KAFKA_OPTS
  value: -javaagent:/kafka/libs/jolokia-jvm-1.6.2-agent.jar=port=8779,host=localhost

The only solution I've found is to use this module:

- module: jolokia
      metricsets: ["jmx"]
      period: 10s
      hosts: ["localhost:8779"]
      namespace: "broker-alt"
      path: "/jolokia/?ignoreErrors=true&canonicalNaming=false"
      jmx.mappings:
        - mbean: 'kafka.network:type=RequestChannel,name=RequestQueueSize'
          attributes:
            - attr: Value
              field: request.channel.queue.size
        - mbean: 'kafka.server:name=FailedProduceRequestsPerSec,type=BrokerTopicMetrics'
          attributes:
            - attr: MeanRate
              field: request.produce.failed_per_second

But why I can't use Kafka module ?

4

This is your problem. Is kafka running? Is the port listening?

5

Hi @Leo-ljr!

Could you provide the specific version/process of your installation so as to try to reproduce it?

For instance, I'm trying with deb version of https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-installation.html#metricbeat-installation and it works on my end.

6

Hi,

Kafka v2.3.1
Metricbeat v7.6.0
Jolokia v1.6.2
K8s v1.7

(The containers of Metricbeat and Kakfka is on the same pod)
Kafka and Metricbeat are deployed with a StatefulSet on 3 nodes
Is that enough, or do we need more details?

And yes Kafka work great

7

It seems that your Kafka/jolokia side is ok since you are able to collect metrics using the jmx metricset.

The original issue you mention, is that exactly what you pointed, the binary you use for some reason does not include the specific metricset (broker).

I tried to reproduce it, installing metricbeat with:

curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.6.2-amd64.deb
sudo dpkg -i metricbeat-7.6.2-amd64.deb

This one worked for me. So could you please provide you installation steps (for Metricbeat)? Thank you!

8

Here is the installation on the docker file:

RUN cd /tmp                                                                                      && \
    wget "https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-$VERSION-amd64.deb" && \
    dpkg -i *.deb                                                                                && \
    rm *.deb

CMD ["/usr/share/metricbeat/bin/metricbeat", "-c", "/etc/metricbeat/metricbeat.yml", "-e"]

Same thing when I use the v7.6.0 or 7.6.2

And the configuration files:

In /etc/metricbeat/metricbeat.yml

metricbeat.yml: |-
    metricbeat.config.modules:
      path: /etc/metricbeat/modules.d/*.yml

      logging.level: debug
      
    output.logstash:
      hosts: '${LOGSTASH_HOSTS}'
      loadbalance: true
      bulk_max_size: 5000
      worker: 1
      index: metricbeat-kafka
      ssl.enabled: '${LOGSTASH_SSL}'
      ssl.certificate_authorities: '${LOGSTASH_SSL_CA}'
      ssl.certificate: '${LOGSTASH_SSL_CRT}'
      ssl.key: '${LOGSTASH_SSL_KEY}'

In /etc/metricbeat/modules.d/

system.yml: |-
    - module: kafka
      metricsets:
        - broker
      period: 10s
      hosts: ["localhost:8779"]
      fields_under_root: true
      fields:
        document_type: metricbeat-kafka-test

Thank you

9

In /etc/metricbeat/modules.d/ you should define the kafka module like:

cat /etc/metricbeat/modules.d/kafka.yml
# Module: kafka
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.6/metricbeat-module-kafka.html
# Metrics collected from a Kafka broker using Jolokia
- module: kafka
  metricsets:
    - broker
  period: 10s
  hosts: ["localhost:8779"]

Your configuration seems to be mixed with that of system module. Not sure if this is the issue yet but we can start from here.

10

Same error

By the way, I didn't specify, but these metricsets work very well:

- module: kafka
      metricsets:
        - partition
        - consumergroup
      period: 10s
      hosts: ["localhost:9092"]
      client_id: metricbeat
      retries: 3
      backoff: 250ms
      topics: []
      fields_under_root: true
      fields:
        document_type: metricbeat-kafka-test
11

This is weird :thinking:.
So VERSION is 7.6.2 right? Cause I'm running manually the very similar installation inside an ubuntu container:

curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.6.2-amd64.deb
sudo dpkg -i metricbeat-7.6.2-amd64.deb

and everything is fine. Could you please verify the VERSION and maybe try to exec inside the container and manually install Metricbeat and give your config a shot?

12

Yes the version is well v7.6.2
Alright i try that
(btw it's a debian 9 container)

13

I used the same image docker I use for Kubernetes and... it works.

root@396c19041214:/etc/metricbeat# metricbeat test modules kafka broker
kafka...
  broker...
    error... ERROR error making http request: Post http://localhost:8779/jolokia/%3FignoreErrors=true&canonicalNaming=false: dial tcp 127.0.0.1:8779: connect: connection refused

And the logs say the same thing:

Error fetching data for metricset kafka.broker: error making http request: Post http://localhost:8779/jolokia/%3FignoreErrors=true&canonicalNaming=false: dial tcp [::1]:8779: connect: cannot assign requested address

This error shows that the module works fine, but why doesn't it work in my K8s environment?

I tried to create a k8s pod with only Metricbeat and its configuration, the error is always the same:

INFO	instance/beat.go:445	metricbeat stopped.
ERROR	instance/beat.go:933	Exiting: 1 error: metricset 'kafka/broker' not found
Exiting: 1 error: metricset 'kafka/broker' not found

This doesn't make any sense

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.