Proper APM setup

[AdamBoi (Oleh Adam Dubnytskyy)]

Kibana version: 7.1.0

Elasticsearch version: 7.1.0

APM Server version: 7.1.0

APM Agent language and version: ruby, ~> 2.8

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
all services (RoR app, APM server, ES node, Kibana service) are running as Docker containers on a single machine (1 vCPU, 3.75 GB memory).

Throughput on RoR application: ~1k rpm

Is there anything special in your setup?
Default setup ended up with APM service "error": "queue is full".

Adjusted configs:
apm-server.yml
queue.mem.events: 8192
output.elasticsearch.bulk_max_size: 5120

Experiment with such configuration ended up with:
"error": "queue is full" and a
fatal error: runtime: out of memory as a final point.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
ES node GC graphs
ES node graphs during experiment
ES node graphs during experiment

Provide logs and/or server output (if relevant):
2019-06-05T12:31:08.508Z ERROR [request] beater/intake_handler.go:86 error handling request {"request_id": "10a0a2a2-6200-40dc-a7c7-c4df48a59709", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "10.128.64.115", "user-agent": "elastic-apm-ruby/2.8.0 http.rb/4.1.1 ruby/2.4.0", "error": "queue is full"}
fatal error: runtime: out of memory

[mikker (Mikkel Malmberg)]

Hi Adam

First off, running all the things on one host of that size is a bit of a stretch. Depending on the amount of memory your app uses, after ES and everything, my guess is that there's not much left for APM Server. Are you doing anything special in your Docker setup to dedicate memory to specific containers?

Either way, APM Server is receiving too many, too large events to keep up.

There's a few things you can do to tweak the event count and size. I'd suggest you look at the agent options stack_trace_limit and transaction_sample_rate.

[AdamBoi (Oleh Adam Dubnytskyy)]

Hello @mikker

All right. Will play around this config options. Thank you!

Also, I'm interested in api_request_time agent option.
It stands for max duration of a single streaming request to APM Server.

Is this streaming request being done as an asynchronous call?
How does it affect the application?

[AdamBoi (Oleh Adam Dubnytskyy)]

tweaking this two options worked for me. Thanks!

[mikker (Mikkel Malmberg)]

Yes, the request happens in a separate thread. The agent tries to do as much as possible off the main thread to let you application do what it does with as little direct overhead as possible.

Happy to hear that you made it work! Thanks for using Elastic APM