Kafka + Metricbeat error:Failed to fetch offset for (topic, partition)

[ekeras (Edgaras Kryževičius)]

I am running elastic stack by this repository. I just changed the network elastic stack is being hosted on, so it would be on the same network as Kafka is.

To set up monitoring kafka metrics with Elastic, I have followed this tutorial.

I have my kafka set up on cloud machine.

Kafka zookeeper and broker:

  zookeeper:
    image: confluentinc/cp-zookeeper:7.0.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      KAFKA_OPTS: "-Dzookeeper.4lw.commands.whitelist=srvr,mntr,ruok,conf,cons"
    labels:
      - co.elastic.metrics/module=zookeeper
      - co.elastic.metrics/hosts=zookeeper:2181
      - co.elastic.metrics/metricsets=mntr,server

  broker:
    image: confluentinc/cp-server:7.0.0
    hostname: broker
    container_name: broker
    volumes:
      - ./jolokia-jvm-1.6.2-agent.jar:/home/appuser/jolokia.jar
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:29092
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: 'true'
      CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'
      KAFKA_JMX_OPTS: '-javaagent:/home/appuser/jolokia.jar=port=9101,host=broker -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false'
    labels:
      - co.elastic.logs/module=kafka
      - co.elastic.metrics/module=kafka
      - co.elastic.metrics/metricsets=partition,consumergroup
      - co.elastic.metrics/hosts='$${data.container.name}:9092'
      - co.elastic.metrics/1.module=kafka
      - co.elastic.metrics/1.metricsets=broker
      - co.elastic.metrics/1.hosts='$${data.container.name}:9101'

Error I am getting from metricbeat:

metricbeat_1     | {"log.level":"error","@timestamp":"2022-05-27T08:13:26.038Z","log.origin":{"file.name":"consumergroup/query.go","file.line":119},"message":"failed to fetch offset for (topic, partition): ('_confluent-metrics', 10)","service.name":"metricbeat","ecs.version":"1.6.0"}
metricbeat_1     | {"log.level":"error","@timestamp":"2022-05-27T08:13:26.040Z","log.origin":{"file.name":"consumergroup/query.go","file.line":119},"message":"failed to fetch offset for (topic, partition): ('_confluent-metrics', 8)","service.name":"metricbeat","ecs.version":"1.6.0"}
metricbeat_1     | {"log.level":"error","@timestamp":"2022-05-27T08:13:26.041Z","log.origin":{"file.name":"consumergroup/query.go","file.line":119},"message":"failed to fetch offset for (topic, partition): ('_confluent-metrics', 7)","service.name":"metricbeat","ecs.version":"1.6.0"}
metricbeat_1     | {"log.level":"error","@timestamp":"2022-05-27T08:13:26.043Z","log.origin":{"file.name":"consumergroup/query.go","file.line":119},"message":"failed to fetch offset for (topic, partition): ('_confluent-metrics', 2)","service.name":"metricbeat","ecs.version":"1.6.0"}