Beats, Kafka output, Azure Event Hubs

Auditbeat and Filebeat seems to be unable to create new topics ("Event Hubs") in an Azure Event Hub Namespace.

Using versions 7.0.0. Azure Event Hub Namespace Standard SKU created, with the Kafka interface enabled. Using the default RootManageSharedAccessKey shared access policy.

The beats can connect to the EHN Kafka interface, but seem to be unable to create a new topic.

Logstash however can create new topics, suggesting the problem's not on the Event Hub side.

Anyone have any ideas?

Example auditbeat config:

  output:
    kafka:
      hosts:
        - "EHN.servicebus.windows.net:9093"
      topic: auditbeat
      version: "2.0.0"
      ssl.enabled: true
      username: "$ConnectionString"
      password: "Endpoint=sb://EHN.servicebus.windows.net/;SharedAccessKeyName=<keyname>;SharedAccessKey=<key>"
      compression: none

Debug output:

2019-04-23T11:02:24.784Z	INFO	kafka/log.go:53	client/metadata fetching metadata for all topics from broker EHN.servicebus.windows.net:9093
2019-04-23T11:02:24.974Z	INFO	kafka/log.go:53	kafka message: Successful SASL handshake
2019-04-23T11:02:24.975Z	INFO	kafka/log.go:53	SASL authentication successful with broker EHN.servicebus.windows.net:9093:4 - [0 0 0 0]
2019-04-23T11:02:24.975Z	INFO	kafka/log.go:53	Connected to broker at EHN.servicebus.windows.net:9093 (unregistered)
2019-04-23T11:02:25.013Z	INFO	kafka/log.go:53	client/brokers registered new broker #0 at EHN.servicebus.windows.net:9093
2019-04-23T11:02:25.013Z	INFO	kafka/log.go:53	kafka message: Successfully initialized new client
2019-04-23T11:02:25.013Z	INFO	pipeline/output.go:105	Connection to kafka(EHN.servicebus.windows.net:9093) established
2019-04-23T11:02:25.013Z	INFO	kafka/log.go:53	client/metadata fetching metadata for [auditbeat] from broker EHN.servicebus.windows.net:9093
2019-04-23T11:02:25.256Z	DEBUG	[processors]	processing/processors.go:183	Publish event: { SNIP }

2019-04-23T11:02:25.262Z	INFO	kafka/log.go:53	kafka message: client/metadata found some partitions to be leaderless
2019-04-23T11:02:25.262Z	INFO	kafka/log.go:53	client/metadata retrying after 250ms... (3 attempts remaining)
2019-04-23T11:02:25.513Z	INFO	kafka/log.go:53	client/metadata fetching metadata for [auditbeat] from broker EHN.servicebus.windows.net:9093
2019-04-23T11:02:25.778Z	INFO	kafka/log.go:53	kafka message: client/metadata found some partitions to be leaderless
2019-04-23T11:02:25.778Z	INFO	kafka/log.go:53	client/metadata retrying after 250ms... (2 attempts remaining)
2019-04-23T11:02:26.028Z	INFO	kafka/log.go:53	client/metadata fetching metadata for [auditbeat] from broker EHN.servicebus.windows.net:9093
2019-04-23T11:02:26.352Z	INFO	kafka/log.go:53	kafka message: client/metadata found some partitions to be leaderless
2019-04-23T11:02:26.352Z	INFO	kafka/log.go:53	client/metadata retrying after 250ms... (1 attempts remaining)
2019-04-23T11:02:26.602Z	INFO	kafka/log.go:53	client/metadata fetching metadata for [auditbeat] from broker EHN.servicebus.windows.net:9093
2019-04-23T11:02:26.844Z	INFO	kafka/log.go:53	kafka message: client/metadata found some partitions to be leaderless
2019-04-23T11:02:26.844Z	INFO	kafka/log.go:53	client/metadata fetching metadata for [auditbeat] from broker EHN.servicebus.windows.net:9093
2019-04-23T11:02:26.844Z	DEBUG	[kafka]	kafka/client.go:251	finished kafka batch
2019-04-23T11:02:26.844Z	DEBUG	[kafka]	kafka/client.go:265	Kafka publish failed with: kafka server: Request was for a topic or partition that does not exist on this broker.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.