How to Improve throughput of Azure EventHub input plugin

Hi Experts, I am using Azure EventHub plugin as input in Logstash, but the event receiving rate is at most around 1.2k/s, it cannot be improved by any methods i tried. Currently the events being ingested to EventHub would be 1.2~1.5k, so there are log delay in our Elasticsearch system.

I tried to set the set larger value of max_batch_size and prefetch_count, original it's 250(max_batch_size )/300 (prefetch_count), i changed them to 1000(max_batch_size )/5000(prefetch_count). nothing happens.

I also tried to scale up the Logstash VM size, but it also does not work and the CPU/JVM usage is around 30%~60%.

So i want to help for this issue.

Here is a config of input plugin:

azure_event_hubs {
threads => 32
max_batch_size => 1000
prefetch_count => 5000
decorate_events => false
type => "test-type"
id => "sample_id"
storage_connection => "DefaultEndp..."
event_hub_connections => ["Endpoint=..."]
consumer_group => "test_consumer"
initial_position => "beginning"

Anyone could help this issue?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.