# Kafka module error
**URL:** https://discuss.elastic.co/t/kafka-module-error/230956
**Category:** Beats
**Tags:** beats-module, metricbeat
**Created:** [May 4, 2020, 11:27am UTC](https://discuss.elastic.co/t/kafka-module-error/230956 "2020-05-04T11:27:29Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@Leo-ljr](https://discuss.elastic.co/u/Leo-ljr)
#### Post date: [May 4, 2020, 11:27am UTC](https://discuss.elastic.co/t/kafka-module-error/230956/1 "2020-05-04T11:27:29Z")
Hi all,
I try to use the producer, consumer and broker metricset for Kafka. But when I use one of them I have this error:
```
2020-05-04T11:06:25.118Z INFO instance/beat.go:445 metricbeat stopped.
2020-05-04T11:06:25.118Z ERROR instance/beat.go:933 Exiting: 1 error: metricset 'kafka/broker' not found
Exiting: 1 error: metricset 'kafka/broker' not found
```
And my config:
```
- module: kafka
metricsets:
- broker
period: 10s
hosts: ["localhost:8779"]
```
It's as if this module doesn't exist in v7.6.0 of metricbeat but the doc says the opposite. Any ideas ?
I'm using version 7.6.0 of metricbeat and version 1.17 of Kubernetes.
---
### Author:  [@Camilo\_Diaz](https://discuss.elastic.co/u/Camilo_Diaz)
#### Post date: [May 5, 2020, 3:38am UTC](https://discuss.elastic.co/t/kafka-module-error/230956/2 "2020-05-05T03:38:23Z")
from the logs, it looks like the metric can't find the broker.
is the port 8779 listening on the server?
---
### Author:  [@Leo-ljr](https://discuss.elastic.co/u/Leo-ljr)
#### Post date: [May 5, 2020, 7:24am UTC](https://discuss.elastic.co/t/kafka-module-error/230956/3 "2020-05-05T07:24:14Z")
Hi,
I'm using this, but I have the same error:
```
- name: KAFKA_OPTS
value: -javaagent:/kafka/libs/jolokia-jvm-1.6.2-agent.jar=port=8779,host=localhost
```
The only solution I've found is to use this module:
```
- module: jolokia
metricsets: ["jmx"]
period: 10s
hosts: ["localhost:8779"]
namespace: "broker-alt"
path: "/jolokia/?ignoreErrors=true&canonicalNaming=false"
jmx.mappings:
- mbean: 'kafka.network:type=RequestChannel,name=RequestQueueSize'
attributes:
- attr: Value
field: request.channel.queue.size
- mbean: 'kafka.server:name=FailedProduceRequestsPerSec,type=BrokerTopicMetrics'
attributes:
- attr: MeanRate
field: request.produce.failed_per_second
```
But why I can't use Kafka module ?
---
### Author:  [@Camilo\_Diaz](https://discuss.elastic.co/u/Camilo_Diaz)
#### Post date: [May 5, 2020, 7:57am UTC](https://discuss.elastic.co/t/kafka-module-error/230956/4 "2020-05-05T07:57:34Z")
This is your problem. Is kafka running? Is the port listening?
> [@Leo-ljr](#):
>
> Exiting: 1 error: metricset 'kafka/broker' not found
---
### Author:  [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)
#### Post date: [May 5, 2020, 8:37am UTC](https://discuss.elastic.co/t/kafka-module-error/230956/5 "2020-05-05T08:37:27Z")
Hi @Leo-ljr!
Could you provide the specific version/process of your installation so as to try to reproduce it?
For instance, I'm trying with `deb` version of [https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-installation.html#metricbeat-installation](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-installation.html#metricbeat-installation) and it works on my end.
---
### Author:  [@Leo-ljr](https://discuss.elastic.co/u/Leo-ljr)
#### Post date: [May 5, 2020, 8:58am UTC](https://discuss.elastic.co/t/kafka-module-error/230956/6 "2020-05-05T08:58:59Z")
Hi,
Kafka v2.3.1
Metricbeat v7.6.0
Jolokia v1.6.2
K8s v1.7
(The containers of Metricbeat and Kakfka is on the same pod)
Kafka and Metricbeat are deployed with a StatefulSet on 3 nodes
Is that enough, or do we need more details?
And yes Kafka work great
---
### Author:  [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)
#### Post date: [May 5, 2020, 9:04am UTC](https://discuss.elastic.co/t/kafka-module-error/230956/7 "2020-05-05T09:04:01Z")
It seems that your Kafka/jolokia side is ok since you are able to collect metrics using the `jmx` metricset.
The original issue you mention, is that exactly what you pointed, the binary you use for some reason does not include the specific metricset (`broker`).
I tried to reproduce it, installing metricbeat with:
```auto
curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.6.2-amd64.deb
sudo dpkg -i metricbeat-7.6.2-amd64.deb
```
This one worked for me. So could you please provide you installation steps (for Metricbeat)? Thank you!
---
### Author:  [@Leo-ljr](https://discuss.elastic.co/u/Leo-ljr)
#### Post date: [May 5, 2020, 9:22am UTC](https://discuss.elastic.co/t/kafka-module-error/230956/8 "2020-05-05T09:22:34Z")
Here is the installation on the docker file:
```
RUN cd /tmp && \
wget "https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-$VERSION-amd64.deb" && \
dpkg -i *.deb && \
rm *.deb
CMD ["/usr/share/metricbeat/bin/metricbeat", "-c", "/etc/metricbeat/metricbeat.yml", "-e"]
```
Same thing when I use the v7.6.0 or 7.6.2
And the configuration files:
In /etc/metricbeat/metricbeat.yml
```
metricbeat.yml: |-
metricbeat.config.modules:
path: /etc/metricbeat/modules.d/*.yml
logging.level: debug
output.logstash:
hosts: '${LOGSTASH_HOSTS}'
loadbalance: true
bulk_max_size: 5000
worker: 1
index: metricbeat-kafka
ssl.enabled: '${LOGSTASH_SSL}'
ssl.certificate_authorities: '${LOGSTASH_SSL_CA}'
ssl.certificate: '${LOGSTASH_SSL_CRT}'
ssl.key: '${LOGSTASH_SSL_KEY}'
```
In /etc/metricbeat/modules.d/
```
system.yml: |-
- module: kafka
metricsets:
- broker
period: 10s
hosts: ["localhost:8779"]
fields_under_root: true
fields:
document_type: metricbeat-kafka-test
```
Thank you
---
### Author:  [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)
#### Post date: [May 5, 2020, 9:32am UTC](https://discuss.elastic.co/t/kafka-module-error/230956/9 "2020-05-05T09:32:27Z")
In `/etc/metricbeat/modules.d/` you should define the `kafka` module like:
```auto
cat /etc/metricbeat/modules.d/kafka.yml
# Module: kafka
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.6/metricbeat-module-kafka.html
# Metrics collected from a Kafka broker using Jolokia
- module: kafka
metricsets:
- broker
period: 10s
hosts: ["localhost:8779"]
```
Your configuration seems to be mixed with that of `system` module. Not sure if this is the issue yet but we can start from here.
---
### Author:  [@Leo-ljr](https://discuss.elastic.co/u/Leo-ljr)
#### Post date: [May 5, 2020, 9:47am UTC](https://discuss.elastic.co/t/kafka-module-error/230956/10 "2020-05-05T09:47:01Z")
Same error
By the way, I didn't specify, but these metricsets work very well:
```
- module: kafka
metricsets:
- partition
- consumergroup
period: 10s
hosts: ["localhost:9092"]
client_id: metricbeat
retries: 3
backoff: 250ms
topics: []
fields_under_root: true
fields:
document_type: metricbeat-kafka-test
```
---
### Author:  [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)
#### Post date: [May 5, 2020, 9:53am UTC](https://discuss.elastic.co/t/kafka-module-error/230956/11 "2020-05-05T09:53:29Z")
This is weird 🤔.
So `VERSION` is `7.6.2` right? Cause I'm running manually the very similar installation inside an ubuntu container:
```auto
curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.6.2-amd64.deb
sudo dpkg -i metricbeat-7.6.2-amd64.deb
```
and everything is fine. Could you please verify the `VERSION` and maybe try to `exec` inside the container and manually install Metricbeat and give your config a shot?
---
### Author:  [@Leo-ljr](https://discuss.elastic.co/u/Leo-ljr)
#### Post date: [May 5, 2020, 9:57am UTC](https://discuss.elastic.co/t/kafka-module-error/230956/12 "2020-05-05T09:57:58Z")
Yes the version is well v7.6.2
Alright i try that
(btw it's a debian 9 container)
---
### Author:  [@Leo-ljr](https://discuss.elastic.co/u/Leo-ljr)
#### Post date: [May 5, 2020, 11:52am UTC](https://discuss.elastic.co/t/kafka-module-error/230956/13 "2020-05-05T11:52:32Z")
I used the same image docker I use for Kubernetes and... it works.
```
root@396c19041214:/etc/metricbeat# metricbeat test modules kafka broker
kafka...
broker...
error... ERROR error making http request: Post http://localhost:8779/jolokia/%3FignoreErrors=true&canonicalNaming=false: dial tcp 127.0.0.1:8779: connect: connection refused
```
And the logs say the same thing:
```
Error fetching data for metricset kafka.broker: error making http request: Post http://localhost:8779/jolokia/%3FignoreErrors=true&canonicalNaming=false: dial tcp [::1]:8779: connect: cannot assign requested address
```
This error shows that the module works fine, but why doesn't it work in my K8s environment?
I tried to create a k8s pod with only Metricbeat and its configuration, the error is always the same:
```
INFO instance/beat.go:445 metricbeat stopped.
ERROR instance/beat.go:933 Exiting: 1 error: metricset 'kafka/broker' not found
Exiting: 1 error: metricset 'kafka/broker' not found
```
This doesn't make any sense
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 2, 2020, 11:52am UTC](https://discuss.elastic.co/t/kafka-module-error/230956/14 "2020-06-02T11:52:37Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.