If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text
Kibana version: 7.7.0
Elasticsearch version: 7.7.0
APM Server version:7.7.0
APM Agent language and version:apm java agent , 1.16
Browser version:chrome
APM-SERVER config:
apm-server:
host: "192.168.2.60:8200"
rum:
enabled: true
event_rate:
limit: 1000
lru_size: 100000
allow_origins : ['*']
ilm:
enabled: "false"
queue:
mem:
events: 4096
flush.min_events: 2048
setup.template.enabled: true
setup.template.name: "apm-%{[observer.version]}"
setup.template.pattern: "apm-%{[observer.version]}-*"
setup.template.fields: "${path.config}/fields.yml"
setup.template.settings:
index:
number_of_shards: 5
codec: best_compression
number_of_routing_shards: 30
mapping.total_fields.limit: 2000
output.elasticsearch:
hosts: ["192.168.2.60:9200"]
compression_level: 1
username: "elastic"
password: "461e.com"
worker: 2
indices:
- index: "apm-%{[observer.version]}-sourcemap"
when.contains:
processor.event: "sourcemap"
- index: "apm-%{[observer.version]}-error-%{+yyyy.MM.dd}"
when.contains:
processor.event: "error"
- index: "apm-%{[observer.version]}-transaction-%{+yyyy.MM.dd}"
when.contains:
processor.event: "transaction"
- index: "apm-%{[observer.version]}-span-%{+yyyy.MM.dd}"
when.contains:
processor.event: "span"
- index: "apm-%{[observer.version]}-metric-%{+yyyy.MM.dd}"
when.contains:
processor.event: "metric"
- index: "apm-%{[observer.version]}-onboarding-%{+yyyy.MM.dd}"
when.contains:
processor.event: "onboarding"
bulk_max_size: 20480
We have put APM into production environment. Recently, we found a serious problem. APM java agent will cause the server memory and CPU to be full when the request is very large.But if apm java agent is not deployed, there will be no such problem.I reproduced this problem in the development environment.
I use the ab command to call my interface crazy.
ab -n 200000 -c 4000 '172.20.43.52:8080/api/v3/market_kline_for_app?symbol=1041&step=15&from=1589421627&to=1589871627'
At this time, it is found that the memory will rise, and then all the memory will be used. The CPU will also be full. Eventually, the service crashes and calls this interface becomes very slow.
I used jmap to dump the memory for analysis.
jmap -dump:live,format=b,file=/home/1.hprof 1
Use mat to analyze the memory. The result is shown in the image.
Memory is used by co.elastic.apm.agent.grpc.helper.GrpcHelperImpl