# Filebeat with event-hub-kafka output, pulish fails: client has run out of available brokers to talk to

**URL:** <https://discuss.elastic.co/t/filebeat-with-event-hub-kafka-output-pulish-fails-client-has-run-out-of-available-brokers-to-talk-to/276263>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 17, 2021, 11:54am UTC](https://discuss.elastic.co/t/filebeat-with-event-hub-kafka-output-pulish-fails-client-has-run-out-of-available-brokers-to-talk-to/276263 "2021-06-17T11:54:34Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Klaus\_zhong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/klaus_zhong/32/90478_2.png) [@Klaus\_zhong](https://discuss.elastic.co/u/Klaus_zhong)\
**Post date:** [June 17, 2021, 11:54am UTC](https://discuss.elastic.co/t/filebeat-with-event-hub-kafka-output-pulish-fails-client-has-run-out-of-available-brokers-to-talk-to/276263/1 "2021-06-17T11:54:34Z")

</div>

# Description

I am using Filebeat to stream my log file to azure event hub and with the configuration as kafka output, the connection to host ip can be established (read from the log below), but when it try to publish topics there is error "Kafka publish failed with: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)". I am sure the setting of connection string and evenhub names, topics are correct cause I test with python script and event hub can receive the messages. When it come to Filebeat, only see requests but no message. Is it due to authentication issues? And my computer need proxy seeting, while there is no "proxy\_url" variable avaliable in output.kafka. I'm using filebeat-7.12.0-windows-x86\_64.

# How to reproduce

The filebeat.yml file looks like:

```auto
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - C:\DataPipeline\DataOut\*.ndjson
  json.key_under_root: true
  json.add_error_key: true
setup.template.settings:
  index.number_of_shards: 1
output.kafka:
  topic: "ehpwt"
  required_acks: 1
  client_id: filebeat
  version: '1.0.0'
  hosts:
    - "ehnamespace.servicebus.windows.net:9093"
  username: "$ConnectionString"
  password: "(The connection string - secure)"
  ssl.enabled: true
  compression: none
  max_message_bytes: 1000000
  partition.round_robin:
    reachable_only: true
logging.level: debug
logging.to_files: true
logging.files:
  path: C:\log\filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

```

and my error log shows:  
2021-06-16T21:21:26.870+0200 INFO [publisher\_pipeline\_output] pipeline/output.go:143 Connecting to kafka([ehnspwt.servicebus.windows.net:9093](http://ehnspwt.servicebus.windows.net:9093))  
2021-06-16T21:21:26.882+0200 DEBUG [kafka] kafka/client.go:100 connect: [[ehnspwt.servicebus.windows.net:9093](http://ehnspwt.servicebus.windows.net:9093)]  
2021-06-16T21:21:26.886+0200 INFO [publisher\_pipeline\_output] pipeline/output.go:151 Connection to kafka([ehnspwt.servicebus.windows.net:9093](http://ehnspwt.servicebus.windows.net:9093)) established  
2021-06-16T21:21:26.891+0200 DEBUG [harvester] log/log.go:107 End of file reached: C:\DataPipeline\DataOut\test01 - Copy (4).ndjson; Backoff now.  
2021-06-16T21:21:27.692+0200 DEBUG [kafka] kafka/client.go:371 finished kafka batch  
2021-06-16T21:21:27.693+0200 DEBUG [kafka] kafka/client.go:385 Kafka publish failed with: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)  
2021-06-16T21:21:27.693+0200 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer  
2021-06-16T21:21:27.701+0200 INFO [publisher] pipeline/retry.go:223 done  
2021-06-16T21:21:28.496+0200 DEBUG [kafka] kafka/client.go:371 finished kafka batch  
2021-06-16T21:21:28.496+0200 DEBUG [kafka] kafka/client.go:385 Kafka publish failed with: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

# Update

An update is that: I tried same setting with filebeat-7.0.0-beta1-windows-x86\_64 which shows totally different logs,

2021-06-17T14:08:03.188+0200 INFO kafka/log.go:53 Failed to connect to broker [ehnspwt.servicebus.windows.net:9093](http://ehnspwt.servicebus.windows.net:9093): dial tcp: lookup [ehnspwt.servicebus.windows.net](http://ehnspwt.servicebus.windows.net): no such host

I tried the same setting with another laptop which can connect directly to internet without proxy and it works, so it's still a proxy problem. Where can I set the similar setting as "proxy\_url: '[http://127.0.0.1:3128](http://127.0.0.1:3128)" in Filebeat with kafka output?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 15, 2021, 1:55pm UTC](https://discuss.elastic.co/t/filebeat-with-event-hub-kafka-output-pulish-fails-client-has-run-out-of-available-brokers-to-talk-to/276263/2 "2021-07-15T13:55:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.