# APM java agent causes my service memory to be full and the service crashes

**URL:** <https://discuss.elastic.co/t/apm-java-agent-causes-my-service-memory-to-be-full-and-the-service-crashes/233654>\
**Category:** APM\
**Tags:** java\
**Created:** [May 21, 2020, 3:31am UTC](https://discuss.elastic.co/t/apm-java-agent-causes-my-service-memory-to-be-full-and-the-service-crashes/233654 "2020-05-21T03:31:08Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![topstargogo](https://avatars.discourse-cdn.com/v4/letter/t/51bf81/32.png) [@topstargogo](https://discuss.elastic.co/u/topstargogo)\
**Post date:** [May 21, 2020, 3:31am UTC](https://discuss.elastic.co/t/apm-java-agent-causes-my-service-memory-to-be-full-and-the-service-crashes/233654/1 "2020-05-21T03:31:08Z")

</div>

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text 🙂

**Kibana version** : 7.7.0

**Elasticsearch version** : 7.7.0

**APM Server version** :7.7.0

**APM Agent language and version** :apm java agent , 1.16

**Browser version** :chrome  
**APM-SERVER config** :

```auto
apm-server:
  host: "192.168.2.60:8200"
  rum:
    enabled: true
    event_rate:
      limit: 1000
      lru_size: 100000
    allow_origins : ['*']
  ilm:
    enabled: "false"
queue:
  mem:
    events: 4096
    flush.min_events: 2048
setup.template.enabled: true
setup.template.name: "apm-%{[observer.version]}"
setup.template.pattern: "apm-%{[observer.version]}-*"
setup.template.fields: "${path.config}/fields.yml"
setup.template.settings:
  index:
    number_of_shards: 5
    codec: best_compression
    number_of_routing_shards: 30
    mapping.total_fields.limit: 2000
output.elasticsearch:
  hosts: ["192.168.2.60:9200"]
  compression_level: 1
  username: "elastic"
  password: "461e.com"
  worker: 2
  indices:
    - index: "apm-%{[observer.version]}-sourcemap"
      when.contains:
        processor.event: "sourcemap"
  
    - index: "apm-%{[observer.version]}-error-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "error"
  
    - index: "apm-%{[observer.version]}-transaction-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "transaction"
  
    - index: "apm-%{[observer.version]}-span-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "span"
  
    - index: "apm-%{[observer.version]}-metric-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "metric"
  
    - index: "apm-%{[observer.version]}-onboarding-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "onboarding"
  bulk_max_size: 20480

```

We have put APM into production environment. Recently, we found a serious problem. APM java agent will cause the server memory and CPU to be full when the request is very large.But if apm java agent is not deployed, there will be no such problem.I reproduced this problem in the development environment.  
I use the ab command to call my interface crazy.

```auto
ab -n 200000 -c 4000 '172.20.43.52:8080/api/v3/market_kline_for_app?symbol=1041&step=15&from=1589421627&to=1589871627'

```

At this time, it is found that the memory will rise, and then all the memory will be used. The CPU will also be full. Eventually, the service crashes and calls this interface becomes very slow.  
I used jmap to dump the memory for analysis.

```auto
 jmap -dump:live,format=b,file=/home/1.hprof 1

```

Use mat to analyze the memory. The result is shown in the image.

 ![1](https://us1.discourse-cdn.com/elastic/original/3X/f/3/f3052fa61a8b8f754dd146a5bac0daedc2347eba.png)  
 ![2](https://us1.discourse-cdn.com/elastic/original/3X/4/4/449fa87098bc33c94825d35e447af42d9ea6e396.png)

Memory is used by co.elastic.apm.agent.grpc.helper.GrpcHelperImpl

---

<div class="post-metadata">

**Author:** ![topstargogo](https://avatars.discourse-cdn.com/v4/letter/t/51bf81/32.png) [@topstargogo](https://discuss.elastic.co/u/topstargogo)\
**Post date:** [May 21, 2020, 3:45am UTC](https://discuss.elastic.co/t/apm-java-agent-causes-my-service-memory-to-be-full-and-the-service-crashes/233654/2 "2020-05-21T03:45:22Z")

</div>

More detailed information

 ![3](https://us1.discourse-cdn.com/elastic/original/3X/2/7/278ea982ce4af8b67dceea9c91a434fb9ee383bb.png)  
 ![4](https://us1.discourse-cdn.com/elastic/original/3X/f/3/f384e23a7319d961483a139877fb3f1538d1ea21.png)

---

<div class="post-metadata">

**Author:** ![topstargogo](https://avatars.discourse-cdn.com/v4/letter/t/51bf81/32.png) [@topstargogo](https://discuss.elastic.co/u/topstargogo)\
**Post date:** [May 21, 2020, 4:12am UTC](https://discuss.elastic.co/t/apm-java-agent-causes-my-service-memory-to-be-full-and-the-service-crashes/233654/3 "2020-05-21T04:12:29Z")

</div>

These occupied memory will not be released even if you stop the stress test. You need to restart the server

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [May 21, 2020, 9:24am UTC](https://discuss.elastic.co/t/apm-java-agent-causes-my-service-memory-to-be-full-and-the-service-crashes/233654/4 "2020-05-21T09:24:11Z")

</div>

Thanks for reporting!

This seems to be related to a gRPC instrumentation related memory leak [issue](https://github.com/elastic/apm-agent-java/issues/1191) we are already aware of and looking into.  
In order to verify, please set the [`disable_instrumentations`](https://www.elastic.co/guide/en/apm/agent/java/current/config-core.html#config-disable-instrumentations) config option to `grpc` and see if the problem is reproduced.  
If not, please watch this GitHub issue, so you get notified once it is resolved.

---

<div class="post-metadata">

**Author:** ![topstargogo](https://avatars.discourse-cdn.com/v4/letter/t/51bf81/32.png) [@topstargogo](https://discuss.elastic.co/u/topstargogo)\
**Post date:** [May 21, 2020, 10:25am UTC](https://discuss.elastic.co/t/apm-java-agent-causes-my-service-memory-to-be-full-and-the-service-crashes/233654/5 "2020-05-21T10:25:49Z")

</div>

Yes, I know this is caused by the call of grpc. If I put

```auto
elastic.apm.disable_instrumentations = grpc 

```

I can’t collect the GRPC information of the service.I just want to collect the GRPC request information of my application

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [May 21, 2020, 2:09pm UTC](https://discuss.elastic.co/t/apm-java-agent-causes-my-service-memory-to-be-full-and-the-service-crashes/233654/6 "2020-05-21T14:09:48Z")

</div>

That's understood, I didn't suggest dropping support for gRPC, as I wrote - we are looking into this. I only asked that you verify that there is no any observable issue when this instrumentation is off.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 11, 2020, 10:10am UTC](https://discuss.elastic.co/t/apm-java-agent-causes-my-service-memory-to-be-full-and-the-service-crashes/233654/7 "2020-06-11T10:10:02Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.