Filebeat. "from kafka to kafka transfer"

Elastic Stack Beats
1

Good day,
I'm trying to set up data transfer from one Kafka to another using Filebeat. The current version does not work with SASL_SSL and I have downloaded 8.0.0-rc1.
Found a new error: "Exiting: module kafka is configured but has no enabled filesets"

Didn't find "filesets" in documentation for version 8.0.0

2

What does ur config look like? Are u using the Kafka module or a Kafka input? It sounds like u want a Kafka input and output.

3

Yes, i want kafka input and output.
My config:

filebeat.config.modules:
  enabled: true
  path: ${path.config}/modules.d/*.yml

filebeat.inputs:
- type: kafka
  hosts:
    - bootstrap-1:9092
  topics: ["test.topic"]
  group_id: "filebeat"
  ssl.enabled: yes
  ssl.certificate_authorities: "/pathTOfile/Root.crt"
  ssl.certificate: "/pathTOfile/Cert.crt"
  ssl.key: "/pathTOfile/Key.rsa.key"
  ssl.key_passphrase: "passphrase"
  kerberos.auth_type: keytab
  kerberos.config_path: "/pathTOfile/krb5.conf"
  kerberos.keytab: "/pathTOfile/mykeytab.keytab"
  kerberos.username: "username"
  kerberos.service_name: "kafka"
  kerberos.realm: "REALM.REALM"


filebeat.modules:
- module: kafka

processors:

output.kafka:
  hosts: ["bootstrap-1:2892"]
  topic: 'test.topic'
  partition.round_robin:
    reachable_only: true
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000
  ssl.enabled: yes
  ssl.certificate_authorities: ["/pathTOfile/Root.crt"]
  ssl.certificate: "/pathTOfile/Cert.crt"
  ssl.key: "/pathTOfile/Key.rsa.key"
  ssl.key_passphrase: "passphrase"
  kerberos.auth_type: keytab
  kerberos.config_path: "/pathTOfile/krb5.conf"
  kerberos.keytab: "/pathTOfile/mykeytab.keytab"
  kerberos.username: "username"
  kerberos.service_name: "kafka"
  kerberos.realm: "REALM.REALM"
4

Ya u don't need the Kafka module. Just the input/output.

5

Thx, working, but got new error:

{"file.name":"kafka/input.go","file.line":152},"message":"Error initializing kafka consumer group"
and
{"message":"kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"}
6

Are the ports for Kafka supposed to be different?

7

Yes, it's two instance of kafka on 1 server. All working, check it

8

It's hard to say. It seems that it's having issues connecting to ur Kafka instance. I'd check all the connection parameters.

9

This connection parameters use in another apps like console-consumer, KafkaHQ, CMAK etc.
Not working only in filebeat ;(

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.