Winlogbeat is not sending logs to new version of Apache Kafka with KRaft

Elastic Stack Beats
, ,
1

Hi everyone,

I'm having a problem with Apache Kafka, related with the configuration of the output format. The arquitecture I created is with docker containers, where Apache Kafka is one and the windows host (virtualized in an debian with qemu) with winlogbeat another container.

My configuration file of winlogbeat is correct when I executed:

.\winlogbeat.exe test config

and when windows host established test connection also is OK.

.\winlogbeat.exe test output

When I started winlogbeat, I see in the Apache Kafka Logs the creation of the topic I refer in

winlogbeat.yml

, but the messages with logs are never sent. I am in a rabbit hole with this.
Also I tried to send messages from a python producer with my host and it works.

Pls help, I provide the configuration and the docker-compose.yml

winlogbeat.yml

 winlogbeat.event_logs:
  - name: Application
    ignore_older: 72h
  - name: Security
  - name: System
  - name: Microsoft-Windows-Windows Defender/Operational
output.kafka:
  hosts: ["192.168.1.47:9094"]
  topic: test
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000
  close_inactive: 7m
processors:
- add_host_metadata:
    netinfo.enabled: true

docker-compose.yml

  kafka1:
    image: confluentinc/cp-kafka:latest
    container_name: kafka1
    hostname: kafka1
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://kafka1:29092,PLAINTEXT_HOST://localhost:9092'
      KAFKA_PROCESS_ROLES: 'broker,controller'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka1:29093'
      KAFKA_LISTENERS: 'CONTROLLER://kafka1:29093,PLAINTEXT://kafka1:29092,PLAINTEXT_HOST://0.0.0.0:9092'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: 'localhost'
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1


      # Replace CLUSTER_ID with a unique base64 UUID using "bin/kafka-storage.sh random-uuid"
      # See https://docs.confluent.io/kafka/operations-tools/kafka-tools.html#kafka-storage-sh
      CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'

the documentation I followed for development are:

https://docs.confluent.io/platform/current/kafka/multi-node.html#connecting-to-ak-on-docker

Why Can’t I Connect to Kafka? | Troubleshoot Connectivity --> Related with docker

thanks for all :smiley:

2

Hi :wave: .

I think you need to change PLAINTEXT_HOST://localhost:9092 in the KAFKA_ADVERTISED_LISTENERS section of your docker compose file to PLAINTEXT_HOST://192.168.1.47:9092

The KAFKA_LISTENERS env var dictates which ports are opened on your container and listening. PLAINTEXT_HOST://0.0.0.0:9092 is fine here as it tells the container to listen on all interfaces.

The KAFKA_ADVERTISED_LISTENERS env variable is what kafka will send to clients after they connect, and the clients will then try to use that broker from that point on.

So by having it set to PLAINTEXT_HOST://localhost:9092, the client will successfully connect at first, but then try to switch over to localhost:9092. Obviously this will fail if not on the same host (which I assume you are not, since the client is Winlogbeat)

I recreated your environment using the winlogbeat config and the docker-compose.yml example you provided. I ended up with the same behavior, in that a connection was established, a topic created, then no events were produced. Then I changed KAFKA_ADVERTISED_LISTENERS to contain a resolvable ip address instead of localhost. After doing so, events started flowing in from Winlogbeat.

(Also, I noticed in your winlogbeat.yml, your host is set to 192.168.1.47:9094. I assume that port is a typo, but just wanted to point it out.)

Hope this helps, please let us know if this fixes your issue :smile:

3

Thank you very much Robin for your reply, I continued researching and thoroughly studying Kafka's documentation and got exactly the same solution you have provided. The key is to understand how Kafka receives the information and from which interface, and how it resolves it internally for the KRaft.
Sorry for replying to you so late, but again thank you very much. From now on when I create a discussion I will visit it more often. :smiley: