# \[ElasticAgent\] failed to publish events: 429 Too Many Requests **URL:** \ **Category:** Elastic Agent\ **Created:** [September 12, 2024, 10:56am UTC](https://discuss.elastic.co/t/elasticagent-failed-to-publish-events-429-too-many-requests/366468 "2024-09-12T10:56:50Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![iamp3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iamp3/32/125051_2.png) [@iamp3](https://discuss.elastic.co/u/iamp3)\ **Post date:** [September 12, 2024, 10:56am UTC](https://discuss.elastic.co/t/elasticagent-failed-to-publish-events-429-too-many-requests/366468/1 "2024-09-12T10:56:50Z")
Hi, I have several ELK clusters deployed in Elastic cloud via Azure integration. We decided to start integrating Azure logs and elastic agents to configure some alerts on the resources we need. On sandbox environment I deployed an agent and configured Azure Logs integration with event-hub+diagnostic settings, then I see that logs are successfully displayed and there are no issues. When trying to configure the same solution on a more heavily loaded environment, I encountered a 429 status code issue. At the moment I have not found any resource spikes on this environment + we are using autoscaling, I also tried processor settings where I set rate\_limit=100/m, which allowed a small part of the logs to reach the elastic, but most of them throw 429 error. ELK and agent version: 8.13.2 Full error log below: ```auto {"log.level":"error","@timestamp":"2024-09-11T11:38:27.919Z","message":"failed to publish events: 429 Too Many Requests: {\"error\":{\"root_cause\":[{\"type\":\"es_rejected_execution_exception\",\"reason\":\"rejected execution of coordinating operation [coordinating_and_primary_bytes=107243174, replica_bytes=0, all_bytes=107243174, coordinating_operation_bytes=6544577, max_coordinating_and_primary_bytes=107374182]\"}],\"type\":\"es_rejected_execution_exception\",\"reason\":\"rejected execution of coordinating operation [coordinating_and_primary_bytes=107243174, replica_bytes=0, all_bytes=107243174, coordinating_operation_bytes=6544577, max_coordinating_and_primary_bytes=107374182]\"},\"status\":429}","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"azure-eventhub-default","type":"azure-eventhub"},"log":{"source":"azure-eventhub-default"},"log.origin":{"file.line":174,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).publishBatch"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","ecs.version":"1.6.0"} {"log.level":"error","@timestamp":"2024-09-11T11:38:28.657Z","message":"failed to perform any bulk index operations: 429 Too Many Requests: {\"error\":{\"root_cause\":[{\"type\":\"es_rejected_execution_exception\",\"reason\":\"rejected execution of coordinating operation [coordinating_and_primary_bytes=107067756, replica_bytes=0, all_bytes=107067756, coordinating_operation_bytes=6544577, max_coordinating_and_primary_bytes=107374182]\"}],\"type\":\"es_rejected_execution_exception\",\"reason\":\"rejected execution of coordinating operation [coordinating_and_primary_bytes=107067756, replica_bytes=0, all_bytes=107067756, coordinating_operation_bytes=6544577, max_coordinating_and_primary_bytes=107374182]\"},\"status\":429}","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"azure-eventhub-default","type":"azure-eventhub"},"log":{"source":"azure-eventhub-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"elasticsearch","log.origin":{"file.line":262,"file.name":"elasticsearch/client.go","function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).publishEvents"},"ecs.version":"1.6.0"} ``` ---
**Author:** ![iamp3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iamp3/32/125051_2.png) [@iamp3](https://discuss.elastic.co/u/iamp3)\ **Post date:** [September 12, 2024, 12:43pm UTC](https://discuss.elastic.co/t/elasticagent-failed-to-publish-events-429-too-many-requests/366468/2 "2024-09-12T12:43:58Z")
I tested with/without coordination node here is the output: 1. With coordination node (2Gb or 4Gb): ```auto {"log.level":"error","@timestamp":"2024-09-12T11:33:55.924Z","message":"failed to publish events: 429 Too Many Requests: {\"error\":{\"root_cause\":[{\"type\":\"es_rejected_execution_exception\",\"reason\":\"rejected execution of coordinating operation [coordinating_and_primary_bytes=214019326, replica_bytes=0, all_bytes=214019326, coordinating_operation_bytes=5949426, max_coordinating_and_primary_bytes=214748364]\"}],\"type\":\"es_rejected_execution_exception\",\"reason\":\"rejected execution of coordinating operation [coordinating_and_primary_bytes=214019326, replica_bytes=0, all_bytes=214019326, coordinating_operation_bytes=5949426, max_coordinating_and_primary_bytes=214748364]\"},\"status\":429}","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"azure-eventhub-default","type":"azure-eventhub"},"log":{"source":"azure-eventhub-default"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":174,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).publishBatch"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"} ``` 1. Without coordination nodes (works better, but data reject error exist): ```auto {"log.level":"error","@timestamp":"2024-09-12T11:54:17.578Z","message":"failed to publish events: 429 Too Many Requests: {\"error\":{\"root_cause\":[{\"type\":\"es_rejected_execution_exception\",\"reason\":\"rejected execution of coordinating operation [coordinating_and_primary_bytes=799758117, replica_bytes=594367, all_bytes=800352484, coordinating_operation_bytes=6404471, max_coordinating_and_primary_bytes=805306368]\"}],\"type\":\"es_rejected_execution_exception\",\"reason\":\"rejected execution of coordinating operation [coordinating_and_primary_bytes=799758117, replica_bytes=594367, all_bytes=800352484, coordinating_operation_bytes=6404471, max_coordinating_and_primary_bytes=805306368]\"},\"status\":429}","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"azure-eventhub-default","type":"azure-eventhub"},"log":{"source":"azure-eventhub-default"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":174,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).publishBatch"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"} or {"log.level":"error","@timestamp":"2024-09-12T12:21:20.214Z","message":"failed to publish events: temporary bulk send failure","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"azure-eventhub-default","type":"azure-eventhub"},"log":{"source":"azure-eventhub-default"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":174,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).publishBatch"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"} or {"log.level":"error","@timestamp":"2024-09-12T12:19:08.074Z","message":"failed to perform any bulk index operations: Post \"https://${EC_DEPLOYMENT_ID}:443/_bulk?filter_path=errors%2Citems.%2A.error%2Citems.%2A.status\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"azure-eventhub-default","type":"azure-eventhub"},"log":{"source":"azure-eventhub-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"elasticsearch","log.origin":{"file.line":262,"file.name":"elasticsearch/client.go","function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).publishEvents"},"ecs.version":"1.6.0"} {"log.level":"error","@timestamp":"2024-09-12T12:19:09.106Z","message":"failed to publish events: Post \"https://${EC_DEPLOYMENT_ID}:443/_bulk?filter_path=errors%2Citems.%2A.error%2Citems.%2A.status\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"azure-eventhub-default","type":"azure-eventhub"},"log":{"source":"azure-eventhub-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","log.origin":{"file.line":174,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).publishBatch"},"ecs.version":"1.6.0"} ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 12, 2024, 12:47pm UTC](https://discuss.elastic.co/t/elasticagent-failed-to-publish-events-429-too-many-requests/366468/3 "2024-09-12T12:47:12Z")
Which version of Elasticsearch are you using? What is the size in terms of nodes and resources of your cluster? How many indices are you actively indexing into? ---
**Author:** ![iamp3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iamp3/32/125051_2.png) [@iamp3](https://discuss.elastic.co/u/iamp3)\ **Post date:** [September 12, 2024, 12:54pm UTC](https://discuss.elastic.co/t/elasticagent-failed-to-publish-events-429-too-many-requests/366468/4 "2024-09-12T12:54:23Z")
Hi @Christian_Dahlqvist, Im using ELK 8.13.2 version. - autoscaling enabled: 1. Elastic - 2 Nodes Hot data: Current: 525 GB storage | 15 GB RAM | 1.9 vCPU Max: 1.03 TB storage | 30 GB RAM | 3.9 vCPU Total (size x zone) 1.03 TB storage | 30 GB RAM | 3.8 vCPU Warm data: 400 GB storage | 2 GB RAM | Up to 2.1 vCPU 800 GB storage | 4 GB RAM | Up to 2.1 vCPU Cold data: 0 MB storage | 0 MB RAM | Up to 0 vCPU 400 GB storage | 2 GB RAM | Up to 2.1 vCPU Frozen data: 0 MB storage | 0 MB RAM | Up to 0 vCPU 6.25 TB storage | 4 GB RAM | Up to 2.1 vCPU Indices ~740, but not sure if all are in active state, could you advice how I could check all active indexing? For Azure Logs integration I see it use data\_stream and create 1 index ---
**Author:** ![iamp3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iamp3/32/125051_2.png) [@iamp3](https://discuss.elastic.co/u/iamp3)\ **Post date:** [September 12, 2024, 2:22pm UTC](https://discuss.elastic.co/t/elasticagent-failed-to-publish-events-429-too-many-requests/366468/5 "2024-09-12T14:22:27Z")
I tried to use output config according this doc: [Elasticsearch output settings | Fleet and Elastic Agent Guide [8.15] | Elastic](https://www.elastic.co/guide/en/fleet/current/es-output-settings.html) looks like it helps a bit, but still observe errors above from time to time. My output config: ```auto bulk_max_size: 4096 worker: 1 queue.mem.events: 8192 queue.mem.flush.min_events: 4096 queue.mem.flush.timeout: 5s compression_level: 1 connection_idle_timeout: 15s ```