Queue is full

Elastic Observability APM
1

Elastic Cloud: 7.14

I have a lot of services that connects to APM and now I have only this message of error:

{"accepted":0,"errors":[{"message":"queue is full"}]}

First of all I want review the messages that we send, but is possible remove all the messages on the queue? I don't care if I lose these messages, I only want start from zero.

Thanks for all!

2

By default apm-server will retry sending queued events to Elasticsearch up to 3 times (see Configure the Elasticsearch output | APM Server Reference [7.14] | Elastic).

After that, the queue should clear out itself. Are you observing different behaviour?

3

Hi!!

I think that the problem is other service that fill de queue :frowning: One service I only have these messages:

[Warning] {"BreakdownMetricsProvider"} The limit of 1000 metricsets has been reached, no new metricsets will be created.

Is possible that this service fill the queue and therefore the other service have this problem?

Thanks for all!

4

The APM Server only holds one internal memory queue. If the queue is filled up by a large amount of events from one service, this can definitely impact other services. It sounds like you already identified the service causing a lot of traffic to the APM Server. Which agent language and version are you using to instrument this service?

5

Hi!!

First of all, thanks for the answer.

I'm using the version 7.14 in the cloud and the differents services has been created in netCore.

I have detect that one of the services only log this error of "The limit of 1000 metricsets has been reached, no new metricsets will be created" but is possible know it 100%. I don't know if I have any index with this information.

Thanks!!

6

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.