# APM with AWS Lambda Java **URL:** \ **Category:** APM\ **Tags:** java\ **Created:** [June 15, 2019, 11:57am UTC](https://discuss.elastic.co/t/apm-with-aws-lambda-java/185936 "2019-06-15T11:57:09Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![f136989c](https://avatars.discourse-cdn.com/v4/letter/f/779978/32.png) [@f136989c](https://discuss.elastic.co/u/f136989c)\ **Post date:** [June 15, 2019, 11:57am UTC](https://discuss.elastic.co/t/apm-with-aws-lambda-java/185936/1 "2019-06-15T11:57:09Z")
Hi, I try to use the apm-java-attach on AWS Lambda openjdk-8 however I am getting "Attaching the Elastic APM Java agent is not possible on this JVM" . Looks like the APM attach is not working on the AWS Lambda OPENJDK 8 ? ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [June 16, 2019, 4:27am UTC](https://discuss.elastic.co/t/apm-with-aws-lambda-java/185936/2 "2019-06-16T04:27:24Z")
Welcome to the forum and thanks for you question! Probably this is because the Java runtime used to run your Lambda is a JRE, rather than a full JDK. In any case, it seems it does not include a `tools.jar` (at least not a valid `AttachProvider`). Is this something you can control? ---
**Author:** ![f136989c](https://avatars.discourse-cdn.com/v4/letter/f/779978/32.png) [@f136989c](https://discuss.elastic.co/u/f136989c)\ **Post date:** [June 16, 2019, 6:41am UTC](https://discuss.elastic.co/t/apm-with-aws-lambda-java/185936/3 "2019-06-16T06:41:01Z")
Hi , thanks for replying. I've sorted out the issue already. Yup, so by default, the AWS Lambda OpenJDK provided by AWS doesn't incl. the tools.jar in the classpath. I have to provide it and then set the property "co.elastic.apm.attach.bytebuddy.agent.toolsjar" with path to the tools.jar. It all works fine now, however, I am having small issues with the reporting. I need to flush all the message / buffers in the agent to the APM server before the Lambda Handler method returns. Are you aware of any way in the agent code that allows me to manually flush all the transaction/span to the APM server? ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [June 16, 2019, 7:38am UTC](https://discuss.elastic.co/t/apm-with-aws-lambda-java/185936/4 "2019-06-16T07:38:36Z")
Yeah, that's another issue we need to see how we tackle with regards to AWS Lambda and the like... Due to the costs of compression and transmission, we send batched events to APM server in the background. In addition, we attach common metadata once per batch, so this is another optimization. So the optimal would be to find a way to utilize the existing communication mechanisms, but that depends on the extent of control you have over the JVM. There is no way to flush events manually, but one thing you can try is set the [`api_request_size` configuration option](https://www.elastic.co/guide/en/apm/agent/java/current/config-reporter.html#config-api-request-time) to a very low value and wait some time **after ending the traced transaction** before letting the Lambda Handler return. This configuration sets the maximum time we allow to accumulate events before sending them. However, note that this may incur considerable overhead, mostly on the JVM but also on the APM server. In general, we made lots of efforts to make sure we don't do any heavy blocking actions on the request-handling thread, and this defies this principal. Just some questions out of curiosity: - Do you intend to do this in a service deployed at production? What volumes are we looking at? - Do yo intend to do this blocking or async in any way? - Did you use the `attach` API as well within the Lambda or as an initiation thing? - What kind of control do you have over the JVM used by AWS to run your Lambdas (eg shared JVM resources, JVM configuration, JVM initiation capabilities)? ---
**Author:** ![f136989c](https://avatars.discourse-cdn.com/v4/letter/f/779978/32.png) [@f136989c](https://discuss.elastic.co/u/f136989c)\ **Post date:** [June 16, 2019, 7:57am UTC](https://discuss.elastic.co/t/apm-with-aws-lambda-java/185936/5 "2019-06-16T07:57:08Z")
Let me answer your questions first:- - Yes. It will be used in a production environment. We are looking at potentially 1k requests per seconds load. - I am currently use the internal config on the Report agent (report\_sync) as this is the only way to have the value reported back to APM server reliably AFAIK. I am not sure how I can optimise it to use async way because as soon as the method returns in the Lambda context, the JVM goes into suspend mode and no living threads will be running until the next request comes in. - I am currently using the attach() mode as I can't use the javaagent on the AWS Lambda. - Not much. It is very limited. I am lucky to figure out there is an option I can use on the agent to set the tools.jar path (co.elastic.apm.attach.bytebuddy.agent.toolsjar) otherwise I won't be able to use APM at all on AWS Lambda. As I mentioned earlier, I am currently using the "report\_sync" config and it seems to be doing the job reporting the value, however it will for sure impact the response time for the APIs (from the client perspective). I will give api\_request\_size a go and see how well it can do. Let me know if you have a better idea. ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [June 16, 2019, 8:20am UTC](https://discuss.elastic.co/t/apm-with-aws-lambda-java/185936/6 "2019-06-16T08:20:35Z")
> [@f136989c](#): > > As I mentioned earlier, I am currently using the "report\_sync" config and it seems to be doing the job reporting the value, however it will for sure impact the response time for the APIs (from the client perspective). Right- this is blocking the response. > [@f136989c](#): > > I will give api\_request\_size a go and see how well it can do I put the right link, but with the wrong config option, I meant `api_request_time`! But note that this will help you only if you can release the response back to the client and then wait within the handler. Otherwise this is essentially blocking as well. > [@f136989c](#): > > I am currently using the attach() mode as I can't use the javaagent on the AWS Lambda But are you using it each time the Lambda is invoked, or as some kind of JVM initiation? ---
**Author:** ![f136989c](https://avatars.discourse-cdn.com/v4/letter/f/779978/32.png) [@f136989c](https://discuss.elastic.co/u/f136989c)\ **Post date:** [June 16, 2019, 8:36am UTC](https://discuss.elastic.co/t/apm-with-aws-lambda-java/185936/7 "2019-06-16T08:36:14Z")
> [@Eyal\_Koren](#): > > But are you using it each time the Lambda is invoked, or as some kind of JVM initiation? The way the AWS Lambda works as the code will init once and stays in memory. So, I the attach() will only happen once as per JVM instance. OK. I will use the api\_request\_time and gives it some times before the method ended and see how it goes. Thanks for your great help in this matter. Very very much appreciated. ---
**Author:** ![f136989c](https://avatars.discourse-cdn.com/v4/letter/f/779978/32.png) [@f136989c](https://discuss.elastic.co/u/f136989c)\ **Post date:** [June 16, 2019, 8:48am UTC](https://discuss.elastic.co/t/apm-with-aws-lambda-java/185936/8 "2019-06-16T08:48:05Z")
My guess is if the APM Reporting is able to allow manual flushing the message to the server , i am sure that will solve my issue. (similar to logger flush method) . I might look at the agent report source code and see if it is doable. ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [June 16, 2019, 9:14am UTC](https://discuss.elastic.co/t/apm-with-aws-lambda-java/185936/9 "2019-06-16T09:14:08Z")
Thanks for the feedback! If that's a valid option (ie can be called without blocking the response) with acceptable overhead - we can certainly consider it (or something of that sort). This may still be used for batching as well, for example- if you invoke your service synthetically once every 10 seconds with a marked request that tells the service to invoke blocking flushes. ---
**Author:** ![f136989c](https://avatars.discourse-cdn.com/v4/letter/f/779978/32.png) [@f136989c](https://discuss.elastic.co/u/f136989c)\ **Post date:** [June 16, 2019, 9:28pm UTC](https://discuss.elastic.co/t/apm-with-aws-lambda-java/185936/10 "2019-06-16T21:28:14Z")
I think we can't assume that the program is able to flush the report on its own thread in the Lambda world. Even with the 'api\_request\_time' approach, there is no 100% guarantee that the thread is able to stay alive to do that task. My preferred approach is to have the reports flush (in blocking) manually if needed. We can do whatever cache/batching we need as usual, but the flush method can just flush all the span/transaction reports in memory to the APM server at will. ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [June 17, 2019, 5:46am UTC](https://discuss.elastic.co/t/apm-with-aws-lambda-java/185936/11 "2019-06-17T05:46:04Z")
I certainly think the Lambda world will require some creativity and adjustments 🙂 > [@f136989c](#): > > We can do whatever cache/batching we need as usual, but the flush method can just flush all the span/transaction reports in memory to the APM server at will. That's what I tried to suggest. I am looking for a way to use the "manual flush" for batching. If you have a way to configure scheduled executions ([maybe using CloudWatch events?](https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/RunLambdaSchedule.html)) - that's great. Otherwise, you will need to cache and flush periodically on a Lambda-executing thread. For that, I can (currently) see two options, both require some logic in the Lambda code: 1. Flush if the decided flush-interval has elapsed since the last flush 2. Use periodic fake requests to your service that will be used for the flush, for example, a request that contains some query parameter telling it to flush and do nothing else The first is easier to implement, but it has no guarantee about flush timings. The second requires, in addition to changes in the service code, something that creates periodic requests. Luckily, we have [Uptime](https://www.elastic.co/blog/elastic-uptime-monitoring-solution-released) 🙂. And maybe- manually flushing after each execution will not be that bad even at 1k requests per second. I will build a quick (and naive) implementation for a blocking manual flush and let's see how it works out. ---
**Author:** ![f136989c](https://avatars.discourse-cdn.com/v4/letter/f/779978/32.png) [@f136989c](https://discuss.elastic.co/u/f136989c)\ **Post date:** [June 17, 2019, 8:40am UTC](https://discuss.elastic.co/t/apm-with-aws-lambda-java/185936/12 "2019-06-17T08:40:03Z")
yup. awaiting for your glory return (of the code change). 😃 ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [June 17, 2019, 10:29am UTC](https://discuss.elastic.co/t/apm-with-aws-lambda-java/185936/13 "2019-06-17T10:29:14Z")
Hi again 🙂 Please download [this API snapshot jar](https://apm-ci.elastic.co/job/apm-agent-java/job/apm-agent-java-mbp/job/PR-672/1/artifact/src/github.com/elastic/apm-agent-java/apm-agent-api/target/apm-agent-api-1.7.1-SNAPSHOT.jar) and [this agent snapshot build](https://apm-ci.elastic.co/job/apm-agent-java/job/apm-agent-java-mbp/job/PR-672/1/artifact/src/github.com/elastic/apm-agent-java/elastic-apm-agent/target/elastic-apm-agent-1.7.1-SNAPSHOT.jar) and try them out. I added a `co.elastic.apm.api.ElasticApm#blockingFlush` API. Note that this is just a quick test build, not implemented in the way we eventually would want to implement that but should be good enough to get some feedback. Will be waiting for your feedback on that. Thanks, Eyal. ---
**Author:** ![f136989c](https://avatars.discourse-cdn.com/v4/letter/f/779978/32.png) [@f136989c](https://discuss.elastic.co/u/f136989c)\ **Post date:** [June 17, 2019, 12:43pm UTC](https://discuss.elastic.co/t/apm-with-aws-lambda-java/185936/14 "2019-06-17T12:43:38Z")
wow... you are super quick!... I will give it a go tomorrow and see how it goes. 🙂 and report it back. stay tuned ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [June 20, 2019, 8:25am UTC](https://discuss.elastic.co/t/apm-with-aws-lambda-java/185936/15 "2019-06-20T08:25:39Z")
Hey! Any updates? Any feedback on this would be very useful for us - basic tracing functionality, overhead, other functionality (eg how does the Metrics tab look and does it makes sense to you?). Thanks, Eyal. ---
**Author:** ![f136989c](https://avatars.discourse-cdn.com/v4/letter/f/779978/32.png) [@f136989c](https://discuss.elastic.co/u/f136989c)\ **Post date:** [June 20, 2019, 8:14pm UTC](https://discuss.elastic.co/t/apm-with-aws-lambda-java/185936/16 "2019-06-20T20:14:37Z")
Hey Eyal, Thanks for keep an eye on this. I have been cop up with production issue in the past few days and haven't got a chance to use the snapshot jar. But i will definitely do it in the weekend and let you know how it goes. ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [July 3, 2019, 5:58am UTC](https://discuss.elastic.co/t/apm-with-aws-lambda-java/185936/17 "2019-07-03T05:58:29Z")
@f136989c Hey! Did you get a chance to test this? ---
**Author:** ![f136989c](https://avatars.discourse-cdn.com/v4/letter/f/779978/32.png) [@f136989c](https://discuss.elastic.co/u/f136989c)\ **Post date:** [July 6, 2019, 10:56pm UTC](https://discuss.elastic.co/t/apm-with-aws-lambda-java/185936/18 "2019-07-06T22:56:10Z")
hey @Eyal_Koren i finally got a chance to test this using the snapshot. It seems to be working fine with the blocking flush. what's the plan now? i am happy to use the snapshot for now. ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [July 7, 2019, 4:35am UTC](https://discuss.elastic.co/t/apm-with-aws-lambda-java/185936/19 "2019-07-07T04:35:30Z")
Hi again Johnny! > [@f136989c](#): > > i am happy to use the snapshot for now. Great, so please do for now. The only limitations are that you won't be able to upgrade until we implement that officially and once we do, you may need to adjust your code to use the new API. > [@f136989c](#): > > what's the plan now? To open a GitHub issue that will include the "official" blocking flush implementation and API. However, I would like this issue to include the documentation of what's needed in order to properly trace AWS Lambda. For that, if you can spend the time to provide the following info, that would be incredible: - Anything special you were required to do in order to make this work- eg anything you did to get the tools.jar available to the agent - The exact settings you used, for example- where and how you added the attach jar and the code to make the attach. - Anything you noticed not working as expected? Specifically- do the metrics make sense? - Anything you think we need to do different for Lambda- eg hide specific metrics, report different metrics etc. - Anything you can say about the overhead of using the agent? The more you elaborate on those, the more you can make this useful for future users following your steps. Thanks a lot for your great feedback!! Eyal. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 28, 2019, 12:35am UTC](https://discuss.elastic.co/t/apm-with-aws-lambda-java/185936/20 "2019-07-28T00:35:40Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.