# I set a large queue value, but apmserver still shows queue is full

**URL:** <https://discuss.elastic.co/t/i-set-a-large-queue-value-but-apmserver-still-shows-queue-is-full/233252>\
**Category:** APM\
**Tags:** server\
**Created:** [May 19, 2020, 6:24am UTC](https://discuss.elastic.co/t/i-set-a-large-queue-value-but-apmserver-still-shows-queue-is-full/233252 "2020-05-19T06:24:02Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [May 19, 2020, 6:24am UTC](https://discuss.elastic.co/t/i-set-a-large-queue-value-but-apmserver-still-shows-queue-is-full/233252/1 "2020-05-19T06:24:02Z")

</div>

apm version：7.6.2  
es version: 7.4.2

When I first saw that queue is full in the log, the queue-related value of apmserver was added, but queue is full will still appear in the log. I don't know what causes apmserver to refuse to receive requests.

 ![FireShot Capture 001 - Stack Monitoring - es - APM - 192.168.10.148](https://us1.discourse-cdn.com/elastic/original/3X/7/b/7b1f8c29bfe174c91b0d829c395ac5944833311e.png)

log file  
[https://send.firefox.com/download/01dbe78edb471ad1/#7Y6\_dS7Rqeri5hdn39ZyQQ](https://send.firefox.com/download/01dbe78edb471ad1/#7Y6_dS7Rqeri5hdn39ZyQQ)

---

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [May 19, 2020, 6:39am UTC](https://discuss.elastic.co/t/i-set-a-large-queue-value-but-apmserver-still-shows-queue-is-full/233252/2 "2020-05-19T06:39:57Z")

</div>

I just tried to delete all indexes of apm- \*, and then apmserver will not prompt "queue is full". Is "queue is full" related to elasticsearch? But I don't see obvious errors on elasticsearch

---

<div class="post-metadata">

**Author:** ![simitt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simitt/32/106406_2.png) [@simitt](https://discuss.elastic.co/u/simitt)\
**Post date:** [May 19, 2020, 11:34am UTC](https://discuss.elastic.co/t/i-set-a-large-queue-value-but-apmserver-still-shows-queue-is-full/233252/3 "2020-05-19T11:34:18Z")

</div>

Hi @wajika, it looks like you might be running into this [APM Server Issue](https://github.com/elastic/apm-server/issues/3698).

---

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [May 20, 2020, 7:56am UTC](https://discuss.elastic.co/t/i-set-a-large-queue-value-but-apmserver-still-shows-queue-is-full/233252/4 "2020-05-20T07:56:48Z")

</div>

@simitt  
I don't understand why I need to delete the alias? What does this have to do with 'queue is full'?  
And if it is a problem caused by the alias, why the alias will conflict?

---

<div class="post-metadata">

**Author:** ![simitt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simitt/32/106406_2.png) [@simitt](https://discuss.elastic.co/u/simitt)\
**Post date:** [May 20, 2020, 9:07am UTC](https://discuss.elastic.co/t/i-set-a-large-queue-value-but-apmserver-still-shows-queue-is-full/233252/5 "2020-05-20T09:07:31Z")

</div>

From the issue:

> apm-server will then do bulk writes to what it thinks is a write alias but instead will `auto_create_index` a real index where the write alias used to be.

The problem you are running into is that the server is writing to an actual index instead of an alias. At some point this leads to an error similar to `Connection marked as failed because the onConnect callback failed: resource 'apm-7.6.2-metric' exists, but it is not an alias`. When this happens the server cannot write any more events to Elasticsearch and the internal memory queue starts filling up. That's when the APM Server starts responding with `queue is full` errors.

Unfortunately there is currently no better solution for recovering from this than described in the [linked APM Server issue](https://github.com/elastic/apm-server/issues/3698).

---

<div class="post-metadata">

**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\
**Post date:** [May 21, 2020, 12:48am UTC](https://discuss.elastic.co/t/i-set-a-large-queue-value-but-apmserver-still-shows-queue-is-full/233252/6 "2020-05-21T00:48:58Z")

</div>

@simitt  
Thank you for your reply, I understand what you mean.  
So the solution is to delete the actual index name?  
Why does this problem occur?

---

<div class="post-metadata">

**Author:** ![simitt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simitt/32/106406_2.png) [@simitt](https://discuss.elastic.co/u/simitt)\
**Post date:** [May 25, 2020, 8:22am UTC](https://discuss.elastic.co/t/i-set-a-large-queue-value-but-apmserver-still-shows-queue-is-full/233252/7 "2020-05-25T08:22:57Z")

</div>

> So the solution is to delete the actual index name?

If you'd like to retain the collected data you can follow [step 2 in the comment](https://github.com/elastic/apm-server/issues/3698#issuecomment-620882797) on how to recover from this issue. Once the cloning finished, you will indeed need to delete the conflicting `index` and ensure the APM Server creates a new `alias`.

> Why does this problem occur?

It can for example occur if one manually deletes the indices after the APM Server took care of the setup. Another possibility is that they were manually setup in the first place and not properly linked with ILM.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 15, 2020, 4:22am UTC](https://discuss.elastic.co/t/i-set-a-large-queue-value-but-apmserver-still-shows-queue-is-full/233252/8 "2020-06-15T04:22:59Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.
