Hi Experts, I am using Azure EventHub plugin as input in Logstash, but the event receiving rate is at most around 1.2k/s, it cannot be improved by any methods i tried. Currently the events being ingested to EventHub would be 1.2~1.5k, so there are log delay in our Elasticsearch system.
I tried to set the set larger value of max_batch_size and prefetch_count, original it's 250(max_batch_size )/300 (prefetch_count), i changed them to 1000(max_batch_size )/5000(prefetch_count). nothing happens.
I also tried to scale up the Logstash VM size, but it also does not work and the CPU/JVM usage is around 30%~60%.
So i want to help for this issue.
Here is a config of input plugin:
azure_event_hubs {
threads => 32
max_batch_size => 1000
prefetch_count => 5000
decorate_events => false
type => "test-type"
id => "sample_id"
storage_connection => "DefaultEndp..."
event_hub_connections => ["Endpoint=..."]
consumer_group => "test_consumer"
initial_position => "beginning"
}