I am using Filebeat to stream my log file to azure event hub and with the configuration as kafka output, the connection to host ip can be established (read from the log below), but when it try to publish topics there is error "Kafka publish failed with: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)". I am sure the setting of connection string and evenhub names, topics are correct cause I test with python script and event hub can receive the messages. When it come to Filebeat, only see requests but no message. Is it due to authentication issues? And my computer need proxy seeting, while there is no "proxy_url" variable avaliable in output.kafka. I'm using filebeat-7.12.0-windows-x86_64.
The filebeat.yml file looks like:
filebeat.inputs: - type: log enabled: true paths: - C:\DataPipeline\DataOut\*.ndjson json.key_under_root: true json.add_error_key: true setup.template.settings: index.number_of_shards: 1 output.kafka: topic: "ehpwt" required_acks: 1 client_id: filebeat version: '1.0.0' hosts: - "ehnamespace.servicebus.windows.net:9093" username: "$ConnectionString" password: "(The connection string - secure)" ssl.enabled: true compression: none max_message_bytes: 1000000 partition.round_robin: reachable_only: true logging.level: debug logging.to_files: true logging.files: path: C:\log\filebeat name: filebeat keepfiles: 7 permissions: 0644
and my error log shows:
2021-06-16T21:21:26.870+0200 INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to kafka(ehnspwt.servicebus.windows.net:9093)
2021-06-16T21:21:26.882+0200 DEBUG [kafka] kafka/client.go:100 connect: [ehnspwt.servicebus.windows.net:9093]
2021-06-16T21:21:26.886+0200 INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to kafka(ehnspwt.servicebus.windows.net:9093) established
2021-06-16T21:21:26.891+0200 DEBUG [harvester] log/log.go:107 End of file reached: C:\DataPipeline\DataOut\test01 - Copy (4).ndjson; Backoff now.
2021-06-16T21:21:27.692+0200 DEBUG [kafka] kafka/client.go:371 finished kafka batch
2021-06-16T21:21:27.693+0200 DEBUG [kafka] kafka/client.go:385 Kafka publish failed with: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
2021-06-16T21:21:27.693+0200 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-06-16T21:21:27.701+0200 INFO [publisher] pipeline/retry.go:223 done
2021-06-16T21:21:28.496+0200 DEBUG [kafka] kafka/client.go:371 finished kafka batch
2021-06-16T21:21:28.496+0200 DEBUG [kafka] kafka/client.go:385 Kafka publish failed with: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
An update is that: I tried same setting with filebeat-7.0.0-beta1-windows-x86_64 which shows totally different logs,
I tried the same setting with another laptop which can connect directly to internet without proxy and it works, so it's still a proxy problem. Where can I set the similar setting as "proxy_url: 'http://127.0.0.1:3128" in Filebeat with kafka output?