# Heavy CPU usage in APM Agents when APM-servers goes down

**URL:** <https://discuss.elastic.co/t/heavy-cpu-usage-in-apm-agents-when-apm-servers-goes-down/164254>\
**Category:** APM\
**Created:** [January 15, 2019, 7:36am UTC](https://discuss.elastic.co/t/heavy-cpu-usage-in-apm-agents-when-apm-servers-goes-down/164254 "2019-01-15T07:36:47Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![pabloborh](https://avatars.discourse-cdn.com/v4/letter/p/3ec8ea/32.png) [@pabloborh](https://discuss.elastic.co/u/pabloborh)\
**Post date:** [January 15, 2019, 7:36am UTC](https://discuss.elastic.co/t/heavy-cpu-usage-in-apm-agents-when-apm-servers-goes-down/164254/1 "2019-01-15T07:36:47Z")

</div>

Hi!

Yesterday i saw in one of my testing environment that agents was consuming a lot of CPU because they was trying to reconnect to APM Server (was down)

Is it possible configure reconnection options to avoid that heavy load?

Thanks!

PD: Java agent v1.3.0

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [January 15, 2019, 7:52am UTC](https://discuss.elastic.co/t/heavy-cpu-usage-in-apm-agents-when-apm-servers-goes-down/164254/2 "2019-01-15T07:52:56Z")

</div>

Hi and thanks a lot for the report.

Do you have any more details like the frequency of reconnections, where the CPU usage exactly stems from and the agent logs? The Java agent uses an increasing backoff so that it only tries to reconnect every 36 seconds after a while.

Thanks,  
Felix

---

<div class="post-metadata">

**Author:** ![pabloborh](https://avatars.discourse-cdn.com/v4/letter/p/3ec8ea/32.png) [@pabloborh](https://discuss.elastic.co/u/pabloborh)\
**Post date:** [January 15, 2019, 8:04am UTC](https://discuss.elastic.co/t/heavy-cpu-usage-in-apm-agents-when-apm-servers-goes-down/164254/3 "2019-01-15T08:04:51Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/f/f/ffc03e7a7d3cf24a5469522211612c4deecaca93.png)

This was the CPU graphic. I'm finding logs...

I'm trying to reproduce a similar scenario.

PD: I see now a different error type. Yesterday i saw

> [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: connect timed out

And today (CPU is OK)

> ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type TRANSACTION with this error: connect timed out

Reconnection time is 36s like you said.

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [January 15, 2019, 8:26am UTC](https://discuss.elastic.co/t/heavy-cpu-usage-in-apm-agents-when-apm-servers-goes-down/164254/4 "2019-01-15T08:26:32Z")

</div>

Are you certain that the high CPU usage was caused by the agent? Or could the cause be somewhere else, maybe because not only the APM Server had an outage.

I just ran some benchmarks without an APM Server and I couldn't reproduce the high CPU usage.

---

<div class="post-metadata">

**Author:** ![pabloborh](https://avatars.discourse-cdn.com/v4/letter/p/3ec8ea/32.png) [@pabloborh](https://discuss.elastic.co/u/pabloborh)\
**Post date:** [January 15, 2019, 8:31am UTC](https://discuss.elastic.co/t/heavy-cpu-usage-in-apm-agents-when-apm-servers-goes-down/164254/5 "2019-01-15T08:31:21Z")

</div>

I can't reproduce the same behaviour, now CPU is fine. Yesterdat APM server was crashing because of Elasticsearch was crashing with the "No shards available or All shards failed" Error.

I'm trying to get same behaviour. When APM Server recovered instantaneously CPU load went down.

Actually I can't do it. Maybe was some side-effect. If i get the same CPU load i will report you.

Thanks for reply and sorry for disturb!

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [January 15, 2019, 8:35am UTC](https://discuss.elastic.co/t/heavy-cpu-usage-in-apm-agents-when-apm-servers-goes-down/164254/6 "2019-01-15T08:35:32Z")

</div>

No worries! Again, thanks for the feedback.

> [@pabloborh](#):
>
> When APM Server recovered instantaneously CPU load went down.

Could the high CPU usage be caused by the Elasticsearch or APM Server pods, rather than your application/the agent?

---

<div class="post-metadata">

**Author:** ![pabloborh](https://avatars.discourse-cdn.com/v4/letter/p/3ec8ea/32.png) [@pabloborh](https://discuss.elastic.co/u/pabloborh)\
**Post date:** [January 15, 2019, 8:39am UTC](https://discuss.elastic.co/t/heavy-cpu-usage-in-apm-agents-when-apm-servers-goes-down/164254/7 "2019-01-15T08:39:47Z")

</div>

> Could the high CPU usage be caused by the Elasticsearch or APM Server pods, rather than your application/the agent?

Pods with heavy CPU load was our java micro-services (they have the agent). It could also be interesting to say that AWS CPU credits was 0 (t2 unlimited was enabled.)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 5, 2019, 4:39am UTC](https://discuss.elastic.co/t/heavy-cpu-usage-in-apm-agents-when-apm-servers-goes-down/164254/8 "2019-02-05T04:39:55Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.
