# 50 percent depreciation in API performace while Using APM agent **URL:** \ **Category:** APM\ **Created:** [January 31, 2019, 9:25am UTC](https://discuss.elastic.co/t/50-percent-depreciation-in-api-performace-while-using-apm-agent/166509 "2019-01-31T09:25:36Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Akash\_Patel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akash_patel/32/39511_2.png) [@Akash\_Patel](https://discuss.elastic.co/u/Akash_Patel)\ **Post date:** [January 31, 2019, 9:25am UTC](https://discuss.elastic.co/t/50-percent-depreciation-in-api-performace-while-using-apm-agent/166509/1 "2019-01-31T09:25:36Z")
Hi, We are recently in a process of adding Elastic APM. But after Performance testing of API with APM agent throughput of API is reduced by 50%. Throughput before adding APM agent - 15200/sec Throughput after adding APM agent - 8100/sec APM-Agent configuration : ```auto logLevel: 'info', serverTimeout: "10s", captureExceptions: true, sourceLinesErrorAppFrames: 5, sourceLinesErrorLibraryFrames: 0, captureErrorLogStackTraces: "messages", captureSpanStackTraces: false, stackTraceLimit: 15, transactionSampleRate:1, captureBody:false, instrument:true, disableInstrumentations:["redis","mysql"], transactionMaxSpans:50, apiRequestTime:"10s", apiRequestSize:"750kb" ``` We tried using different configurations but there wasn't much improvement in throughput. Tried lowering `transactionSampleRate` , `apiRequestTime` and `apiRequestSize` but insignificant improvement. Do we need to tune apm-server configurations? Is there anything else need to be tuned in APM agent? Please let me know if you need more info on this. **Kibana version** : 6.5.4 **Elasticsearch version** : 6.5.4 **APM Server version** : 6.5.4 **APM Agent language and version** : Node js, 2.1.0 Thanks ---
**Author:** ![digitalron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/digitalron/32/38224_2.png) [@digitalron](https://discuss.elastic.co/u/digitalron)\ **Post date:** [January 31, 2019, 1:17pm UTC](https://discuss.elastic.co/t/50-percent-depreciation-in-api-performace-while-using-apm-agent/166509/2 "2019-01-31T13:17:39Z")
Have you tried scaling out Elastic APM? Intake API bandwidth can be a problem given the quantities above. Assuming a 10% sampling rate, that is still about 800 transactions you are sending per second, easily exceeding 800 kB. Also, do you see CPU or memory-bound bottlenecks on the instrumented services? You might want to play around with those settings. ---
**Author:** ![Akash\_Patel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akash_patel/32/39511_2.png) [@Akash\_Patel](https://discuss.elastic.co/u/Akash_Patel)\ **Post date:** [February 1, 2019, 7:29am UTC](https://discuss.elastic.co/t/50-percent-depreciation-in-api-performace-while-using-apm-agent/166509/3 "2019-02-01T07:29:29Z")
Hi, Thanks for your time For now we are running APM server on single machine with 40 cores, 512GB of ram with only single instance of Apm server. I think you are talking about [this](https://www.elastic.co/guide/en/apm/server/current/tune-apm-server.html#add-apm-server-instances) parameter right ? You said scaling out so should i try multiple instances of apm server on same machine? or each instance running on different machine? I thought of running multiple instances of apm server on same machine and load balancing them with NGINX. will this help? Thanks ---
**Author:** ![digitalron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/digitalron/32/38224_2.png) [@digitalron](https://discuss.elastic.co/u/digitalron)\ **Post date:** [February 1, 2019, 9:18am UTC](https://discuss.elastic.co/t/50-percent-depreciation-in-api-performace-while-using-apm-agent/166509/4 "2019-02-01T09:18:15Z")
You can try multiple instances of the APM server on the same machine and load balance them in front. I've done it with Haproxy in front of three dockerised APM Servers on a single server and it improved out APM intake throughput. It also helped that we set up a cluster of Elasticsearch stores: 3 Master, 4 Data. Keep monitoring the resources and see if you are CPU or memory or network bound. In our case, we did a bit of custom ingest + Geo and Useragent so CPU use was a bit high. ---
**Author:** ![Akash\_Patel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akash_patel/32/39511_2.png) [@Akash\_Patel](https://discuss.elastic.co/u/Akash_Patel)\ **Post date:** [February 1, 2019, 11:54am UTC](https://discuss.elastic.co/t/50-percent-depreciation-in-api-performace-while-using-apm-agent/166509/5 "2019-02-01T11:54:37Z")
Thanks for your guidance. Will try the same, and also monitor CPU+Mem usage. Meanwhile, Can you provide me some suggestions to begin with the scaling ? Assuming that machine has 40 core and 512GB of RAM. - How many instances of APM server should i run to begin with? - What should be the values of following properties in YAML file for each instance to begin with? `queue.mem.events`, `output.elastic.bulk_max_size` , `output.elastic.workers` - Do i need to change any other property in YAML file in order to optimize apm server ? Thanks ---
**Author:** ![digitalron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/digitalron/32/38224_2.png) [@digitalron](https://discuss.elastic.co/u/digitalron)\ **Post date:** [February 1, 2019, 1:42pm UTC](https://discuss.elastic.co/t/50-percent-depreciation-in-api-performace-while-using-apm-agent/166509/6 "2019-02-01T13:42:04Z")
To be honest, the quantity would really depend on the amount and type of instrumentation you are doing. Trial and error is the way to go. Start small, run the load test, then observe CPU and RAM along with I/O. On my laptop (32 GB RAM MacBook 2018), I have 3 APM Servers running with 2 GB heap each and a 7-node Elasticsearch cluster. I can easily ingest a combined 4000 calls per minute of a fairly complex transaction (distributed across six Java services) at 100% sampling rate. ---
**Author:** ![Akash\_Patel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akash_patel/32/39511_2.png) [@Akash\_Patel](https://discuss.elastic.co/u/Akash_Patel)\ **Post date:** [February 1, 2019, 3:57pm UTC](https://discuss.elastic.co/t/50-percent-depreciation-in-api-performace-while-using-apm-agent/166509/7 "2019-02-01T15:57:11Z")
Sure. Will try it out. Thanks ---
**Author:** ![stephenbelanger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenbelanger/32/34880_2.png) [@stephenbelanger](https://discuss.elastic.co/u/stephenbelanger)\ **Post date:** [February 1, 2019, 6:28pm UTC](https://discuss.elastic.co/t/50-percent-depreciation-in-api-performace-while-using-apm-agent/166509/8 "2019-02-01T18:28:21Z")
I would turn the sample rate _way_ down. A good balance is aiming for 50-100 traces per second. Setting `sourceLinesErrorAppFrames` to `0` and `captureErrorLogStackTraces` to `'never'` might also help. ---
**Author:** ![Akash\_Patel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akash_patel/32/39511_2.png) [@Akash\_Patel](https://discuss.elastic.co/u/Akash_Patel)\ **Post date:** [February 2, 2019, 6:32am UTC](https://discuss.elastic.co/t/50-percent-depreciation-in-api-performace-while-using-apm-agent/166509/9 "2019-02-02T06:32:46Z")
Sure will try this out and let you know. Thanks ---
**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\ **Post date:** [February 19, 2019, 8:29am UTC](https://discuss.elastic.co/t/50-percent-depreciation-in-api-performace-while-using-apm-agent/166509/10 "2019-02-19T08:29:26Z")
Did you try out the suggestions? Did it help to retain the throughput? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 12, 2019, 4:29am UTC](https://discuss.elastic.co/t/50-percent-depreciation-in-api-performace-while-using-apm-agent/166509/11 "2019-03-12T04:29:26Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.