# APM Server timing out under heavy load **URL:** \ **Category:** APM\ **Tags:** server\ **Created:** [October 5, 2020, 9:17am UTC](https://discuss.elastic.co/t/apm-server-timing-out-under-heavy-load/250956 "2020-10-05T09:17:26Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![PedroMSantosD](https://avatars.discourse-cdn.com/v4/letter/p/c6cbf5/32.png) [@PedroMSantosD](https://discuss.elastic.co/u/PedroMSantosD)\ **Post date:** [October 5, 2020, 9:17am UTC](https://discuss.elastic.co/t/apm-server-timing-out-under-heavy-load/250956/1 "2020-10-05T09:17:26Z")
Category: -\> TCP Traffic / Dimensioning. Problem: Nodes are not responding TCP duriing peak hours (self recovery, affects inbound traffic thought) **Kibana version** : (not used for config) **Elasticsearch version** : Currently writing to logstash: 7.5.1 **APM Server version** : 7.5.1 **APM Agent language and version** : (several). java agents mainly, plus latest RUM agents. **Browser version** : (several). **Original install method (e.g. download page, yum, deb, from source, etc.) and version**: # rpm -qa |grep apm apm-server-7.5.1-1.x86\_64 **Fresh install or upgraded from other version?** (upgraded) **Is there anything special in your setup?** For example, are you using the Logstash or Kafka outputs? load balancer -\> APM-Servers -\> Logstash -\> Elasticsearch Load Balancer infront of APM server Farm (8 nodes). Load Balancer does a tCP-half open health check that fails during peak hours. Icinga monitoring checks on TCP status of port 8200 of each node. During Peak hours we have both, TCP timeouts on apm servers and large delays on tcp responses from all of the farm members (see tcp stats below) APM servers write to Logstash nodes directly **Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**: Current Behaviour; during peak load hour, I get TCP Timeouts on port 8200, losing and unknown amount of RUM and APM traffic. I do not get any indicators of the actual problem as CPU user time remains below 20%, memory footprint is the same, regardless of the traffic, TCP timeouts are not logged. Expected behaviour: (a) Information on if there is further fine tuning possible, as There are No CPU / Memory issues on nodes (see graphs belows), and apm limits have been tuned as per [apm recomendations](https://www.elastic.co/guide/en/apm/server/current/tune-apm-server.html) There are no memory or CPU overloads on the APM Farm during peak hours. APM Traffic exists, but lots(some ??) of it is not received at server due to TCP time outs ![apm-server-metrics](https://us1.discourse-cdn.com/elastic/original/3X/2/e/2e656d2f4b4fcc58ec71b6c321243db55c3017d3.png) ![rum and apm transactions logged to elasticsearch](https://us1.discourse-cdn.com/elastic/original/3X/2/e/2e56a56afa896e5b853dbce99c416c28fea526c0.png) **Steps to reproduce** : 1. Heavy application load causes it on my infrastructure. No issues on low load conditions. 2. 3. **Errors in browser console (if relevant)**: **Provide logs and/or server output (if relevant)**: Logs from peak hours reflecting errors (one node only): > root@xxx# cat apm-server.log | sed "s/.\*response\_code// " | cut -d: -f2 | sed "s/}//" | sort | uniq -c > 875 200 > 44697 202 > 55 401, "error" > 106 403, "error" > 4 429, "error" > 1 500, "error" > root@xxx# cat apm-server.log.1 | sed "s/.\*response\_code// " | cut -d: -f2 | sed "s/}//" | sort | uniq -c > 2498 200 > 127482 202 > 143 401, "error" > 245 403, "error" > 9 429, "error" > 5 500, "error" 401 and 403 errors have been checked, there are configuration issues on some of the agent (no keys, no autoconfig from kibana enabled) but expected. 429 errors have been trobleshooted by expanding limits of event rate on node configuation: > #---------------------------- APM Server - RUM Real User Monitoring ---------------------------- > > # Enable Real User Monitoring (RUM) Support. By default RUM is disabled. > > rum: > enabled: true > > ``` > event_rate: > limit: 1000 > lru_size: 8000 > > allow_origins: [ '... > > ``` sample 500 Errors: > apm-server.log:2020-10-05T04:47:07.947-0400 ERROR [request] middleware/log\_middleware.go:74 internal error {"request\_id": "9a79369b-7989-45be-924c-793c95e53e31", "method": "POST", "URL": "/intake/v2/rum/events", "content\_length": 11974, "remote\_address": "xxx.xxx.xxx.xxx", "user-agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 13\_7 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.2 Mobile/15E148 Safari/604.1", "response\_code": 500, "error": "read tcp xxx.xxx.xxx.xxx:8200-\>xxx.xxx.xxx.xxx:35968: read: connection reset by peer"} > > 2020-10-05T04:02:33.922-0400 ERROR [request] middleware/log\_middleware.go:74 internal error {"request\_id": "1653857a-3855-40e5-821c-542fbb9e3325", "method": "POST", "URL": "/intake/v2/rum/events", "content\_length": 8196, "remote\_address": "xxx.xxx.xxx.xxx", "user-agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 14\_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Mobile/15E148 Safari/604.1", "response\_code": 500, "error": "read tcp xxx.xxx.xxx.xxx:8200-\>xxx.xxx.xxx.xxx:19925: read: connection reset by peer"} > 2020-10-05T04:24:43.255-0400 ERROR [request] middleware/log\_middleware.go:74 internal error {"request\_id": "5ee51918-1cef-4618-b585-8b5da34482a1", "method": "POST", "URL": "/intake/v2/rum/events", "content\_length": 3726, "remote\_address": "xxx.xxx.xxx.xxx", "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10\_15\_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1 Safari/605.1.15", "response\_code": 500, "error": "read tcp xxx.xxx.xxx.xxx:8200-\>xxx.xxx.xxx.xxx:6896: i/o timeout"} > 2020-10-05T04:32:24.261-0400 ERROR [request] middleware/log\_middleware.go:74 internal error {"request\_id": "8e8a62d5-7a8e-453d-b846-001fcd692116", "method": "POST", "URL": "/intake/v2/rum/events", "content\_length": 5608, "remote\_address": "xxx.xxx.xxx.xxx", "user-agent": "Mozilla/5.0 (Linux; Android 9) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/87.0.4263.2 Mobile DuckDuckGo/5 Safari/537.36", "response\_code": 500, "error": "context canceled"} Can you tell if there is a limit on how many TCP connections can a single node handle, as to adjust dimensionting of the farm? Are threre any default process limits that might need overriding? Each node is running 4 cures, but CPU load and CPU user time points to releasing 2 cores in favour of expanding size fam. Node Limits > # ulimit -a > > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 15066 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 4096 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) unlimited > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 26, 2020, 5:17am UTC](https://discuss.elastic.co/t/apm-server-timing-out-under-heavy-load/250956/2 "2020-10-26T05:17:41Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.