# Enhancing Ingestion Rate of Elastic Agent Reading from Azure Event Hub **URL:** \ **Category:** Elastic Agent\ **Tags:** filebeat, metricbeat\ **Created:** [December 18, 2024, 12:55pm UTC](https://discuss.elastic.co/t/enhancing-ingestion-rate-of-elastic-agent-reading-from-azure-event-hub/372162 "2024-12-18T12:55:36Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![RajuParipelly](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rajuparipelly/32/117473_2.png) [@RajuParipelly](https://discuss.elastic.co/u/RajuParipelly)\ **Post date:** [December 18, 2024, 12:55pm UTC](https://discuss.elastic.co/t/enhancing-ingestion-rate-of-elastic-agent-reading-from-azure-event-hub/372162/1 "2024-12-18T12:55:36Z")
I have successfully deployed an Elastic Agent that interfaces with an Azure Event Hub. The current configuration processes approximately 5 to 10 million logs per hour, with individual log sizes ranging between 5 to 20 KB. The system utilizes a virtual machine (VM) to run the Elastic Agent and subsequently transmit the logs to Elastic Cloud. However, I've encountered an issue where the agent does not ingest data as quickly as it's generated by the source, resulting in a consistent lag. Upon reviewing the VM's resources, I observed that the CPU utilization is between 20-40%, and there is ample memory available. Despite these seemingly sufficient resources, the lag persists. Could you provide guidance on how to enhance the ingestion rate of the Elastic Agent to better match the pace of data generation from the Azure Event Hub? ---
**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\ **Post date:** [December 18, 2024, 1:53pm UTC](https://discuss.elastic.co/t/enhancing-ingestion-rate-of-elastic-agent-reading-from-azure-event-hub/372162/2 "2024-12-18T13:53:32Z")
The first thing to try would be to make sure you're running at least 8.12 Agent and to switch that particular agent to using the throughput preset see: [Using Elastic Agent Performance Presets in 8.12 | Elastic Blog](https://www.elastic.co/blog/using-elastic-agent-performance-presets-in-8-12) Beyond that, would you be able to share your integration configuration for the event hub integration as well as outline for us any additional processors or ingest pipelines you may have configured? Can you also share a snippet from your agent log so I can see the internal metrics being generated for this input on your agent? As a general recommendation for pub/sub integrations like event hub, we recommend employing multiple smaller nodes to scale throughout, see our recommendations for AWS s3/SQS which will apply to eventhub as well: [Get the most from Elastic Agent with Amazon S3 and SQS | Elastic Blog](https://www.elastic.co/blog/elastic-agent-amazon-s3-sqs) ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [December 18, 2024, 2:21pm UTC](https://discuss.elastic.co/t/enhancing-ingestion-rate-of-elastic-agent-reading-from-azure-event-hub/372162/3 "2024-12-18T14:21:03Z")
Besides what @strawgate mentioned, you may need to check on Azure side if there is any throttling. Azure may throttle your requests depending on some factors, like event rate, and event size and the number of TUs you have. ---
**Author:** ![RajuParipelly](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rajuparipelly/32/117473_2.png) [@RajuParipelly](https://discuss.elastic.co/u/RajuParipelly)\ **Post date:** [December 18, 2024, 2:52pm UTC](https://discuss.elastic.co/t/enhancing-ingestion-rate-of-elastic-agent-reading-from-azure-event-hub/372162/4 "2024-12-18T14:52:07Z")
Hi @strawgate - We are using Elastic Agent 8.17 version and below are the performance tuning we have applied bulk\_max\_size: 4096 worker: 16 queue.mem.events: 131072 queue.mem.flush.min\_events: 4096 queue.mem.flush.timeout: 5s compression\_level: 1 idle\_connection\_timeout: 15s And also we have an ingest pipeline used to parse the logs (Contains some grok patterns, and json parsing as well) below are the metrics of the elastic agent ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/0/0027a9edcada99c1c6079c16063641505692c31e.png)