# Beats, Kafka output, Azure Event Hubs

**URL:** <https://discuss.elastic.co/t/beats-kafka-output-azure-event-hubs/178020>\
**Category:** Beats\
**Created:** [April 23, 2019, 11:24am UTC](https://discuss.elastic.co/t/beats-kafka-output-azure-event-hubs/178020 "2019-04-23T11:24:58Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rob3](https://avatars.discourse-cdn.com/v4/letter/r/a4c791/32.png) [@Rob3](https://discuss.elastic.co/u/Rob3)\
**Post date:** [April 23, 2019, 11:24am UTC](https://discuss.elastic.co/t/beats-kafka-output-azure-event-hubs/178020/1 "2019-04-23T11:24:59Z")

</div>

Auditbeat and Filebeat seems to be unable to create new topics ("Event Hubs") in an Azure Event Hub Namespace.

Using versions 7.0.0. Azure Event Hub Namespace Standard SKU created, with the Kafka interface enabled. Using the default RootManageSharedAccessKey shared access policy.

The beats can connect to the EHN Kafka interface, but seem to be unable to create a new topic.

Logstash however can create new topics, suggesting the problem's not on the Event Hub side.

Anyone have any ideas?

**Example auditbeat config:**

```auto
  output:
    kafka:
      hosts:
        - "EHN.servicebus.windows.net:9093"
      topic: auditbeat
      version: "2.0.0"
      ssl.enabled: true
      username: "$ConnectionString"
      password: "Endpoint=sb://EHN.servicebus.windows.net/;SharedAccessKeyName=<keyname>;SharedAccessKey=<key>"
      compression: none

```

**Debug output:**

```auto
2019-04-23T11:02:24.784Z	INFO	kafka/log.go:53	client/metadata fetching metadata for all topics from broker EHN.servicebus.windows.net:9093
2019-04-23T11:02:24.974Z	INFO	kafka/log.go:53	kafka message: Successful SASL handshake
2019-04-23T11:02:24.975Z	INFO	kafka/log.go:53	SASL authentication successful with broker EHN.servicebus.windows.net:9093:4 - [0 0 0 0]
2019-04-23T11:02:24.975Z	INFO	kafka/log.go:53	Connected to broker at EHN.servicebus.windows.net:9093 (unregistered)
2019-04-23T11:02:25.013Z	INFO	kafka/log.go:53	client/brokers registered new broker #0 at EHN.servicebus.windows.net:9093
2019-04-23T11:02:25.013Z	INFO	kafka/log.go:53	kafka message: Successfully initialized new client
2019-04-23T11:02:25.013Z	INFO	pipeline/output.go:105	Connection to kafka(EHN.servicebus.windows.net:9093) established
2019-04-23T11:02:25.013Z	INFO	kafka/log.go:53	client/metadata fetching metadata for [auditbeat] from broker EHN.servicebus.windows.net:9093
2019-04-23T11:02:25.256Z	DEBUG	[processors]	processing/processors.go:183	Publish event: { SNIP }

2019-04-23T11:02:25.262Z	INFO	kafka/log.go:53	kafka message: client/metadata found some partitions to be leaderless
2019-04-23T11:02:25.262Z	INFO	kafka/log.go:53	client/metadata retrying after 250ms... (3 attempts remaining)
2019-04-23T11:02:25.513Z	INFO	kafka/log.go:53	client/metadata fetching metadata for [auditbeat] from broker EHN.servicebus.windows.net:9093
2019-04-23T11:02:25.778Z	INFO	kafka/log.go:53	kafka message: client/metadata found some partitions to be leaderless
2019-04-23T11:02:25.778Z	INFO	kafka/log.go:53	client/metadata retrying after 250ms... (2 attempts remaining)
2019-04-23T11:02:26.028Z	INFO	kafka/log.go:53	client/metadata fetching metadata for [auditbeat] from broker EHN.servicebus.windows.net:9093
2019-04-23T11:02:26.352Z	INFO	kafka/log.go:53	kafka message: client/metadata found some partitions to be leaderless
2019-04-23T11:02:26.352Z	INFO	kafka/log.go:53	client/metadata retrying after 250ms... (1 attempts remaining)
2019-04-23T11:02:26.602Z	INFO	kafka/log.go:53	client/metadata fetching metadata for [auditbeat] from broker EHN.servicebus.windows.net:9093
2019-04-23T11:02:26.844Z	INFO	kafka/log.go:53	kafka message: client/metadata found some partitions to be leaderless
2019-04-23T11:02:26.844Z	INFO	kafka/log.go:53	client/metadata fetching metadata for [auditbeat] from broker EHN.servicebus.windows.net:9093
2019-04-23T11:02:26.844Z	DEBUG	[kafka]	kafka/client.go:251	finished kafka batch
2019-04-23T11:02:26.844Z	DEBUG	[kafka]	kafka/client.go:265	Kafka publish failed with: kafka server: Request was for a topic or partition that does not exist on this broker.

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 21, 2019, 1:25pm UTC](https://discuss.elastic.co/t/beats-kafka-output-azure-event-hubs/178020/2 "2019-05-21T13:25:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.