# APM Server queue is full

**URL:** <https://discuss.elastic.co/t/apm-server-queue-is-full/266866>\
**Category:** APM\
**Tags:** server\
**Created:** [March 10, 2021, 8:53pm UTC](https://discuss.elastic.co/t/apm-server-queue-is-full/266866 "2021-03-10T20:53:43Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![rp346](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rp346/32/53882_2.png) [@rp346](https://discuss.elastic.co/u/rp346)\
**Post date:** [March 10, 2021, 8:53pm UTC](https://discuss.elastic.co/t/apm-server-queue-is-full/266866/1 "2021-03-10T20:53:43Z")

</div>

I have setup APM Server along with EBK stack. APM server has following configuration

> **apm-server.yml**
>
> ```auto
> apm-server:
> host: "0.0.0.0:8200"
> frontend:
> enabled: false
> setup.template.settings:
> index:
> number_of_shards: 1
> codec: best_compression
> setup.dashboards.enabled: false
> setup.kibana:
> host: "kibana:5601"
> output.elasticsearch:
> hosts: ["elasticsearch:9200"]
> indices:
> - index: "apm-%{[beat.version]}-sourcemap"
> when.contains:
> processor.event: "sourcemap"
> - index: "apm-%{[beat.version]}-error-%{+yyyy.MM.dd}"
> when.contains:
> processor.event: "error"
> - index: "apm-%{[beat.version]}-transaction-%{+yyyy.MM.dd}"
> when.contains:
> processor.event: "transaction"
> - index: "apm-%{[beat.version]}-span-%{+yyyy.MM.dd}"
> when.contains:
> processor.event: "span"
> 
> ```

Following deployment manifest

> **Deployment**
>
> ```auto
> apiVersion: apps/v1
> kind: Deployment
> metadata:
> name: apm-server
> namespace: kube-logging
> labels:
> app: apm-server
> spec:
> replicas: 2
> strategy:
> type: RollingUpdate
> rollingUpdate:
> maxSurge: 1
> maxUnavailable: 1
> selector:
> matchLabels:
> app: apm-server
> template:
> metadata:
> labels:
> app: apm-server
> spec:
> containers:
> - name: apm-server
> image: docker.elastic.co/apm/apm-server:7.11.0
> ports:
> - containerPort: 8200
> name: apm-port
> volumeMounts:
> - name: apm-server-config
> mountPath: /usr/share/apm-server/apm-server.yml
> readOnly: true
> subPath: apm-server.yml
> volumes:
> - name: apm-server-config
> configMap:
> name: apm-server-config
> 
> ```

Application is pumping 2K TPU to APM server which causing Internal queue full errors at application side. I want to setup Horizontal Pod Autoscaling (HPA) for APM Server in Kubernetes, for which I need to understand how can I detect when internal queue is full.  
Any help is appreciated.

**Kibana version** : `docker.elastic.co/kibana/kibana:7.10.1`

**Elasticsearch version** : `docker.elastic.co/elasticsearch/elasticsearch:7.10.1`

**APM Server version** : `docker.elastic.co/apm/apm-server:7.11.0`

**APM Agent language and version** : `^5.10.1` (Python)

**Original install method (e.g. download page, yum, deb, from source, etc.) and version**: `Kubernetes`

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [March 15, 2021, 2:27am UTC](https://discuss.elastic.co/t/apm-server-queue-is-full/266866/2 "2021-03-15T02:27:39Z")

</div>

Before scaling out the server, I would recommend going through [Common problems | APM Server Reference [7.11] | Elastic](https://www.elastic.co/guide/en/apm/server/current/common-problems.html#queue-full) if you haven't already, and tune the server according to [Tune APM Server | APM Server Reference [7.11] | Elastic](https://www.elastic.co/guide/en/apm/server/current/tune-apm-server.html#tune-output-config)

For information about the internal queue, you can make use of the stack monitoring features: [Monitor APM Server | APM Server Reference [7.11] | Elastic](https://www.elastic.co/guide/en/apm/server/current/monitoring.html). The specific metrics you'll want to look at are `libbeat.pipeline.*`. The number of "queue is full" responses sent by the server is also captured in `apm-server.profile.response.errors.queue`.

You will probably need to write code to expose these to HPA. There's an open request to expose these metrics for Prometheus that you might want to upvote and watch: [Feature: export prometheus metrics for monitoring/autoscaling · Issue #4299 · elastic/apm-server · GitHub](https://github.com/elastic/apm-server/issues/4299)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 4, 2021, 10:28pm UTC](https://discuss.elastic.co/t/apm-server-queue-is-full/266866/3 "2021-04-04T22:28:41Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.