# APM queue is full (output Kafka) **URL:** \ **Category:** APM\ **Tags:** server\ **Created:** [December 2, 2020, 3:49pm UTC](https://discuss.elastic.co/t/apm-queue-is-full-output-kafka/257382 "2020-12-02T15:49:41Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![YvorL](https://avatars.discourse-cdn.com/v4/letter/y/9fc348/32.png) [@YvorL](https://discuss.elastic.co/u/YvorL)\ **Post date:** [December 2, 2020, 3:49pm UTC](https://discuss.elastic.co/t/apm-queue-is-full-output-kafka/257382/1 "2020-12-02T15:49:41Z")
Hi, What can I do if the APM server (7.9.2) continuously logs 503 errors (queue is full)? I checked the tuning options but for many reasons my output isn't Elasticsearch but Kafka and I couldn't find any pointer that would help pushing more data to my Kafka cluster. Thanks! ---
**Author:** ![simitt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simitt/32/106406_2.png) [@simitt](https://discuss.elastic.co/u/simitt)\ **Post date:** [December 2, 2020, 4:20pm UTC](https://discuss.elastic.co/t/apm-queue-is-full-output-kafka/257382/2 "2020-12-02T16:20:22Z")
Hi @YvorL, similar to the config options for Elasticsearch there is a [`bulk_max_size`](https://www.elastic.co/guide/en/apm/server/current/kafka-output.html#_bulk_max_size_3) and a [`bulk_flush_frequency`](https://www.elastic.co/guide/en/apm/server/current/kafka-output.html#_bulk_flush_frequency) option available for Kafka. The config options for [adjusting the APM Server internal queue](https://www.elastic.co/guide/en/apm/server/current/tune-apm-server.html#adjust-queue-size) are independent of the actual output. Seeing `503` on a regular basis might just be an indicator though that your APM Server cannot keep up with the load and you might actually need to add resources to the server. ---
**Author:** ![YvorL](https://avatars.discourse-cdn.com/v4/letter/y/9fc348/32.png) [@YvorL](https://discuss.elastic.co/u/YvorL)\ **Post date:** [December 3, 2020, 3:29pm UTC](https://discuss.elastic.co/t/apm-queue-is-full-output-kafka/257382/3 "2020-12-03T15:29:08Z")
Thanks! I've added a new node and all 503 errors disappeared. Now I see a couple of "too large body" (400) entries per minute. Sometimes paired with connection reset. ``` ERROR [request] middleware/log_middleware.go:95 request body too large {"request_id": "---", "method": "POST", "URL": "---", "content_length": 813677, "remote_address": "---", "user-agent": "---", "response_code": 400, "error": "event exceeded the permitted size."} ERROR [request] middleware/log_middleware.go:95 request body too large {"request_id": "---", "method": "POST", "URL": "---", "content_length": 2021384, "remote_address": "---", "user-agent": "---", "response_code": 400, "error": "event exceeded the permitted size., event exceeded the permitted size., event exceeded the permitted size., event exceeded the permitted size."} ERROR [request] request/context.go:173 write response {"request_id": "---", "method": "POST", "URL": "---", "content_length": 2021384, "remote_address": "---", "user-agent": "---", "error": "write tcp XYZ:X->XYZ:X: write: connection reset by peer"} ``` Could you help me out how to solve this error? Thanks! ---
**Author:** ![simitt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simitt/32/106406_2.png) [@simitt](https://discuss.elastic.co/u/simitt)\ **Post date:** [December 3, 2020, 4:34pm UTC](https://discuss.elastic.co/t/apm-queue-is-full-output-kafka/257382/4 "2020-12-03T16:34:47Z")
The APM Server has an `apm-server.max_event_size` config option that defaults to `307200` bytes, which you could increase. Usually I would not expect APM events to exceed this size though. Could you provide information about which APM agent you are using that leads to these errors, and also agent side custom configuration? ---
**Author:** ![YvorL](https://avatars.discourse-cdn.com/v4/letter/y/9fc348/32.png) [@YvorL](https://discuss.elastic.co/u/YvorL)\ **Post date:** [December 3, 2020, 5:33pm UTC](https://discuss.elastic.co/t/apm-queue-is-full-output-kafka/257382/5 "2020-12-03T17:33:10Z")
It's a custom agent so that's on me, also I overlooked that setting in the [docs](https://www.elastic.co/guide/en/apm/server/current/configuration-process.html#max_event_size) too. Thanks again, you helped a lot! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 24, 2020, 1:33pm UTC](https://discuss.elastic.co/t/apm-queue-is-full-output-kafka/257382/6 "2020-12-24T13:33:14Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.