Filebeat connect with Kafka Kerberos(SASL_SSL) not working

Elastic Stack Beats
1

Hi - I am using Filebeat 7.9 version. using filebeat, read data from log file and push to kafka topic. our Kafka use SASL_SSL with Kerberos mechanism. when I set up with below parms, it errors out.

kerberos.enabled: "true"
  kerberos.auth_type: keytab
  kerberos.config_path: krb5.conf
  kerberos.realm: ""
  kerberos.service_name: "kafka"
  kerberos.username: "kafka"
  kerberos.keytab: keytab path
  
  ssl.enabled: "true"
  ssl.certificate_authorities: cer file
  ssl.certificate: pem file
  ssl.key: key file

Below is the error we get. Any help appreciated. Thanks.

DEBUG [harvester] log/log.go:107 End of file reached: E:\Logs\file.log; Backoff now.
DEBUG [kafka] kafka/client.go:277 finished kafka batch
DEBUG [kafka] kafka/client.go:291 Kafka publish failed with: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

Kafka publish failed with: circuit breaker is open

1 Like
2

Could you please share the debug logs of Kafka?

3

I dont see any logs in broker end. I dont see parameter to provide JAAS conf file. I just give SSL and KRB5 conf here. Am I missing something from output.kafka parameters.

4

I am able to connect logstash to kafka(with kerberos). I have given JAAS conf and krb5 conf along with SSL truststore and keystore. For Filebeat, I get "client has run out of available brokers to talk to" and "circuit breaker is open" error. I assume, between filebeat and logstash, logstash might consume more memory than filebeat and so we dont want to go with logstash. Processing wise we just read logs and send to kafka. we dont do any aggregation or processing.

5

Hi, I am using filebeat 7.7.1 with kafka 0.10.0.0, and meet with the same problem.
This occurs when I use the kerberos configuration.
Filebeat is logging as below infinitely:

DEBUG [kafka] kafka/client.go:276 finished kafka batch
DEBUG [kafka] kafka/client.go:290 Kafka publish failed with: circuit breaker is open
INFO [publisher] pipeline/retry.go:196 retryer: send unwait-signal to consumer
INFO [publisher] pipeline/retry.go:198 done
INFO [publisher] pipeline/retry.go:173 retryer: send wait signal to consumer
INFO [publisher] pipeline/retry.go:175 done

6

Is kerberos authentication based on x-pack? I use filebeat-oss-7.7.1.

7

Hi, i'am use filebeat 7.9.2 and have some issue with kerberos.

2020-09-28T08:42:13.039Z DEBUG [kafka] kafka/client.go:291 Kafka publish failed with: circuit breaker is open
2020-09-28T08:42:13.039Z DEBUG [kafka] kafka/client.go:277 finished kafka batch
2020-09-28T08:42:13.039Z DEBUG [kafka] kafka/client.go:291 Kafka publish failed with: circuit breaker is open
2020-09-28T08:42:13.039Z INFO [publisher] pipeline/retry.go:213 retryer: send wait signal to consumer
2020-09-28T08:42:13.039Z INFO [publisher] pipeline/retry.go:217 done

My config:

    - type: container
      enabled: true
      paths:
        - /var/log/containers/*project-prod*.log
      processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          matchers:
          - logs_path:
              logs_path: "/var/log/containers/"

    logging:
      level: info
      selectors: ['*']
      to_files: false
      to_syslog: false

    filebeat.modules:
    - module: kafka

    output.kafka:
      enabled: true
      hosts: ["kafka1.domain.ru:9092","kafka2.domain.ru:9092","kafka3.domain.ru:9092"]
      topic: "grace-log"
      #partition.round_robin:
      #  reachable_only: false
      #required_acks: 1
      #compression: gzip
      kerberos.enabled: true
      kerberos.auth_type: keytab
      kerberos.username: "svc_user"
      kerberos.keytab: "/tmp/keytab"
      kerberos.service_name: "kafka"
      kerberos.config_path: "/tmp/krb5.conf"
      kerberos.realm: "DOMAIN.RU"

Is there a solution to this problem ?

Thank you.

8

It is also part of the oss version.

9

Please also share the debug logs of Kafka. Without looking at those, I cannot tell what is going wrong exactly.

Also, which version are you using?

10

Hi, problem was on server side. More not actual.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.