# Why is there a lot of "Waiting 30s" in the service log? **URL:** \ **Category:** APM\ **Tags:** dotnet\ **Created:** [November 9, 2021, 8:29am UTC](https://discuss.elastic.co/t/why-is-there-a-lot-of-waiting-30s-in-the-service-log/288722 "2021-11-09T08:29:42Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\ **Post date:** [November 9, 2021, 8:29am UTC](https://discuss.elastic.co/t/why-is-there-a-lot-of-waiting-30s-in-the-service-log/288722/1 "2021-11-09T08:29:42Z")
> Waiting 30s... Wait time is taken from max-age directive in Cache-Control header in APM Server's response. dbgIterationsCount: 37436. > Waiting 30s... Wait time is taken from max-age directive in Cache-Control header in APM Server's response. dbgIterationsCount: 36. > Waiting 30s... Wait time is taken from max-age directive in Cache-Control header in APM Server's response. dbgIterationsCount: 27. Today, when I checked the logs collected by filebeat, I found that many service logs showed "wait" messages. Why? I did not find a solution in "Troubleshoot Common problems" ---
**Author:** ![riferrei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/riferrei/32/104867_2.png) [@riferrei](https://discuss.elastic.co/u/riferrei)\ **Post date:** [November 9, 2021, 2:49pm UTC](https://discuss.elastic.co/t/why-is-there-a-lot-of-waiting-30s-in-the-service-log/288722/2 "2021-11-09T14:49:48Z")
I think you should read this discussion here from last year: > [@DotNetAgent: No data has been received from agents yet](https://discuss.elastic.co/t/dotnetagent-no-data-has-been-received-from-agents-yet/241056): > > Kibana version: 7.8.0 Elasticsearch version: 7.8.0 APM Server version: 7.8.0 APM Agent language and version: .NET (1.6.0) Browser version: Version 78.0.3904.97 (Official Build) (64-bit) Original install method (e.g. download page, yum, deb, from source, etc.) and version: Elastic Search Website Fresh install or upgraded from other version? Fresh Install Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in fron… — @riferrei ---
**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\ **Post date:** [November 10, 2021, 5:43am UTC](https://discuss.elastic.co/t/why-is-there-a-lot-of-waiting-30s-in-the-service-log/288722/3 "2021-11-10T05:43:41Z")
> [@riferrei](#): > > I think you should read this discussion here from last year: Thank you for your reply. I read this discussion and didn't understand that my error has any connection with it. What I want to confirm is what does "Waiting 30s" mean? The apm-server queue is full? ---
**Author:** ![riferrei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/riferrei/32/104867_2.png) [@riferrei](https://discuss.elastic.co/u/riferrei)\ **Post date:** [November 10, 2021, 1:10pm UTC](https://discuss.elastic.co/t/why-is-there-a-lot-of-waiting-30s-in-the-service-log/288722/4 "2021-11-10T13:10:12Z")
It may be, but we can't know for sure without proper debugging. If you could, please run the APM Server with the options `-e -d "*"` to increase verboseness in the details of the log. Also, check if there is a load balancer in-between the agent and the APM server — sometimes, these timeouts may be caused by unbalanced timeouts. ---
**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\ **Post date:** [November 10, 2021, 2:31pm UTC](https://discuss.elastic.co/t/why-is-there-a-lot-of-waiting-30s-in-the-service-log/288722/5 "2021-11-10T14:31:50Z")
For debugging, it may be a little troublesome, the apm-server is running on the kubernetes cluster. The agent reaches the apm-server through the request domain name (nginx forwarding), I checked the nginx log, and there is no exception. Next, I will spend a little time observing. ---
**Author:** ![riferrei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/riferrei/32/104867_2.png) [@riferrei](https://discuss.elastic.co/u/riferrei)\ **Post date:** [November 10, 2021, 3:14pm UTC](https://discuss.elastic.co/t/why-is-there-a-lot-of-waiting-30s-in-the-service-log/288722/6 "2021-11-10T15:14:48Z")
Ah, that's interesting. Check the timeout configured for NGINX and the APM Server. Ideally, your load balancer timeout should be between the agent timeout and the APM Server timeout — and certainly smaller than the APM Server. For example: APM agent ( **10s** ) ➡ Load Balancer ( **15s** ) ➡ APM Server ( **30s** ) — @riferrei ---
**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\ **Post date:** [November 11, 2021, 3:37am UTC](https://discuss.elastic.co/t/why-is-there-a-lot-of-waiting-30s-in-the-service-log/288722/7 "2021-11-11T03:37:20Z")
I intercepted a piece of debug log, there is "done send ack" in the log, it is obvious that the data has been sent to es, but no new data appeared on es. [https://paste.ubuntu.com/p/kNt9BWVdvV/](https://paste.ubuntu.com/p/kNt9BWVdvV/) ---
**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\ **Post date:** [November 12, 2021, 6:47am UTC](https://discuss.elastic.co/t/why-is-there-a-lot-of-waiting-30s-in-the-service-log/288722/8 "2021-11-12T06:47:43Z")
Paste the service log [https://paste.ubuntu.com/p/yhXJBJTdwM/](https://paste.ubuntu.com/p/yhXJBJTdwM/) ---
**Author:** ![riferrei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/riferrei/32/104867_2.png) [@riferrei](https://discuss.elastic.co/u/riferrei)\ **Post date:** [November 12, 2021, 6:20pm UTC](https://discuss.elastic.co/t/why-is-there-a-lot-of-waiting-30s-in-the-service-log/288722/9 "2021-11-12T18:20:25Z")
According to your service log, this message represents the .NET agent trying to poll data from the APM Server and timing out. > `dbgIterationsCount` represents the number of attempts to complete the data polling but failed. Each attempt increments the counter and then schedules another attempt. This means that something is going on between the agent and the APM Server at a network level. If I were you; I would start capturing some network packets from this communication to understand further what is happening. As you can see in the agent's code, it is a simple request-reply HTTP interaction. — @riferrei ---
**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\ **Post date:** [November 13, 2021, 2:16am UTC](https://discuss.elastic.co/t/why-is-there-a-lot-of-waiting-30s-in-the-service-log/288722/10 "2021-11-13T02:16:27Z")
Okay, I will try to capture and analyze. ---
**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\ **Post date:** [November 15, 2021, 2:26am UTC](https://discuss.elastic.co/t/why-is-there-a-lot-of-waiting-30s-in-the-service-log/288722/11 "2021-11-15T02:26:13Z")
I checked the captured data, but I don't know how to analyze it. Can you take a look? [https://drive.google.com/file/d/1ulOd19IN4a5YE3IbeSz82zaIAb0Cqkqo](https://drive.google.com/file/d/1ulOd19IN4a5YE3IbeSz82zaIAb0Cqkqo) > **[111.rar](https://drive.google.com/file/d/1z9CrpuLJgeyKraBPYOvVIQWGC2TmAH7q/edit)** > > Google Drive file. ---
**Author:** ![wajika](https://avatars.discourse-cdn.com/v4/letter/w/977dab/32.png) [@wajika](https://discuss.elastic.co/u/wajika)\ **Post date:** [November 17, 2021, 1:24am UTC](https://discuss.elastic.co/t/why-is-there-a-lot-of-waiting-30s-in-the-service-log/288722/12 "2021-11-17T01:24:31Z")
I guess this problem is caused by an incorrect mapping. I added a field to the template. Because this field has multiple formats (such as string, json), Elasticsearch did not map it correctly. Apm-server sent it to The es message should not have received a successful signal, so it has been waiting in a loop. The above is based on my guess. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 7, 2021, 9:24pm UTC](https://discuss.elastic.co/t/why-is-there-a-lot-of-waiting-30s-in-the-service-log/288722/13 "2021-12-07T21:24:59Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.