Hello,
We are trying to use APM to monitor our website but so far APM starts producing 503 Queue is full error after some time. After this happens it won't get back to normal, only restart of the APM service helps. The server is literally sleeping, CPU usage was around 15% and memory only 50% full. When I enabled it today at night, I also did performance tests and there was no problem with 900rpm but then it crashed at around 50rpm... All the performance settings seems to be useless. I don't think our traffic is so big that 4 CPUs and 12GB (half-used) can't handle it.
It throws this error no matter how big/moderate/conservative the values in configuration are...
Kibana version: 7.4
Elasticsearch version: 7.4
APM Server version: 7.4
Original install method (e.g. download page, yum, deb, from source, etc.) and version: Official 7.x repository
Fresh install or upgraded from other version? Upgraded from 7.2 before using APM
Is there anything special in your setup? No additional outputs except Elasticsearch
I left monitoring turned on, because it didn't crash for the first time so we knew what was going on:
Hello @simitt, I am sorry, but I could not find anything in the logs from the time of crash. APM does not keep logs at all and Elasticsearch logs are almost empty except some unrelated stuff.
Do you mean that you have disabled keeping logs? By default the APM Server does write to log files.
I assume you might encounter a similar bug to what we have seen in another discuss entry (APM Failed to publish events: temporary bulk send failure / Queue is full 503 error).
From 7.4 on apm pipelines are enabled by default, and a new field client.ip is indexed. Providing invalid data for fields that are part of the pipelines, can lead to errors and ingestion retries. This seems to happen in some cases for 7.4 for the client.ip field. There is a bug fix for this, that will be part of the next patch release for 7.4.
Until then I suggest you disable the pipeline, and remove the client.ip field from being ingested. You can do so by changing your apm-servery.yml file to include following settings:
@simitt Any ETA for that patch? We have the same issue but unfortunately cannot apply that workaround because Elastic Cloud doesn't allow these settings to be set.
@dnorth98 7.4.1 including the patch was released today, you can enable the pipeline again and get rid of dropping the field. Thanks for confirming it works as expected @rocketleap.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.