Logstash Kafka output - Event Hub Namespace limit results in "UNKNOWN_TOPIC_OR_PARTITION" errors

Elastic Stack Logstash
1

Dear Logstash enthousiasts,

I run a cluster of 4 Logstash servers (v8.14) with about 10 Logstash pipelines per node. I use keepalived as VIP failover method. So all 4 nodes have the same 10 Logstash pipelines configured. All pipelines have a Azure DCR and Kafka (to Event Hub) output configured for log ingestion into Sentinel (LAW) and ADX (by Event Hub). My current Event Hub Namespace reached the limit of 10 Event Hubs. When I try to setup a new pipeline (ex. eventhub-02) with a Kafka output to an EH on the new EH namespace, I run into “UNKNOWN_TOPIC_OR_PARTITION” errors with this new Kafka output.

I setup a similar setup in our Test environment to reproduce the problem. When checking the logstash service startup log (see below). Although the eventhub-02 Kafka output is configured with a different connection string to the 2nd EH namespace. The Kafka Producer clientId is linked to the 1st EH namespace for the metadata fetch. Which results in the “UNKNOWN_TOPIC_OR_PARTITION“ errors as the eventhub-02 metadata is not fetched from the correct EH namespace. Did someone run into the same issue and how can this be solved to have kafka outputs on a Logstash server to 2 different EH namespaces?

I need to mention that I already reproduced this issue with the latest 8.x and 9.x Logstash version that contained recenter Kafka output module versions.

Thanks for your comments and input.

Regards,

Nico

Logstash log extract:

[2025-09-03T10:40:08,653][INFO ][logstash.agent ] Pipelines running {:count=>3, :running_pipelines=>[:eventhubtest2, :eventhubtest1, :"metricbeat-logstash-sentinel"], :non_running_pipelines=>}

[2025-09-03T10:40:08,678][INFO ][org.apache.kafka.clients.Metadata][eventhubtest1] [Producer clientId=eventhub-01] Cluster ID: logstash-ehn-01.servicebus.windows.net

[2025-09-03T10:40:08,681][INFO ][org.apache.kafka.clients.Metadata][metricbeat-logstash-sentinel] [Producer clientId=logstash-metricbeat] Cluster ID: logstash-ehn-01.servicebus.windows.net

[2025-09-03T10:40:08,715][INFO ][org.logstash.beats.Server][metricbeat-logstash-sentinel][metricbeat_logstash] Starting server on port: 5044

[2025-09-03T10:40:08,812][INFO ][org.apache.kafka.clients.Metadata][eventhubtest2] [Producer clientId=eventhub-02] Cluster ID: logstash-ehn-01.servicebus.windows.net

[2025-09-03T10:40:10,640][WARN ][org.apache.kafka.clients.NetworkClient][eventhubtest2] [Producer clientId=eventhub-02] Error while fetching metadata with correlation id 3 : {logstash-eventhub-02=UNKNOWN_TOPIC_OR_PARTITION}

[2025-09-03T10:40:12,172][WARN ][org.apache.kafka.clients.NetworkClient][eventhubtest2] [Producer clientId=eventhub-02] Error while fetching metadata with correlation id 4 : {logstash-eventhub-02=UNKNOWN_TOPIC_OR_PARTITION}

[2025-09-03T10:40:13,764][WARN ][org.apache.kafka.clients.NetworkClient][eventhubtest2] [Producer clientId=eventhub-02] Error while fetching metadata with correlation id 5 : {logstash-eventhub-02=UNKNOWN_TOPIC_OR_PARTITION}

[2025-09-03T10:40:15,311][WARN ][org.apache.kafka.clients.NetworkClient][eventhubtest2] [Producer clientId=eventhub-02] Error while fetching metadata with correlation id 6 : {logstash-eventhub-02=UNKNOWN_TOPIC_OR_PARTITION}

[2025-09-03T10:40:16,848][WARN ][org.apache.kafka.clients.NetworkClient][eventhubtest2] [Producer clientId=eventhub-02] Error while fetching metadata with correlation id 7 : {logstash-eventhub-02=UNKNOWN_TOPIC_OR_PARTITION}

2

It is not clear what you mean with this.

Can you share the outputs you have, redacting the sensitive information but leaving enough information to identify what is the same and what is not the same in the outputs?

3

Hi @leandrojmp ,

Thanks for your reply.

Let me explain, I have 2 Logstash pipelines with a Kafka output to different Azure EH Namespaces and a dedicated jaas config containing the connection string for the related EH Namespace. Below the log entries, I observerd in the logstash plain log file, when starting the Logstash service hosting both pipelines. So the Logstash is linking the second pipeline to the first EH namespace instead of the second EH namespace (that is configured in the jaas) and tries to fetch the eventhub-02 metadata from the first EH Namespace, which results in the “Unknow topic or partition error”.

  • Feedback from Elastic support:

It is possible that the underlying Apache Kafka client used by the Logstash output is not strictly pipeline-isolated and cached metadata may bleed from the first initialized eventhub to the other.

  • Log extract:

[2025-08-25T11:19:41,635][INFO ][org.apache.kafka.clients.Metadata][eventhubtest1] [Producer clientId=eventhub-01] Cluster ID: logstashehnamespace-01.servicebus.windows.net

[2025-08-25T11:19:41,727][INFO ][org.apache.kafka.clients.Metadata][ eventhubtest2] [Producer clientId=eventhub-02] Cluster ID: logstashehnamespace-01.servicebus.windows.net

Regards,

Nico

4

Can you share more about your configuration. For example, are you using the jaas_path option?

5

I don't think the client id will make much difference as this is mostly used to help track the source on kafka logs.

You need to share your configurations a asked, it is pretty complicated to try to provide any insight without seeing the configuration.