Not able connect to elastic apm server from AWS lambda

Pratiksha_Nagoshe · August 24, 2022, 1:07pm

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text

TIP 1: select at least one tag that further categorizes your topic. For example server for APM Server related questions, java for questions regarding the Elastic APM Java agent, or ui for questions about the APM App within Kibana.

TIP 2: Check out the troubleshooting guide first. Not only will it help you to resolve common problems faster but it also explains in more detail which information we need before we can properly help you.

Kibana version:

Elasticsearch version: 7.15.0

APM Server version: 7.15.0

APM Agent language and version: java and elastic-apm-extension-ver-1-0-2-x86_64 , elastic-apm-java-ver-1-33-0

Browser version:

Original install method (e.g. download page, yum, deb, from source, etc.) and version:

Fresh install or upgraded from other version?

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.
load balancer is there
Description of the problem including expected versus actual behaviour. Please include screenshots (if relevant):
not getting hit to elastic apm from aws lambda , added environment variables as well
|ELASTIC_APM_LAMBDA_APM_SERVER|http://14.142.143.138:8200|
|ELASTIC_APM_LOG_LEVEL|TRACE|
|ELASTIC_APM_SERVICE_NAME|service_name|

Steps to reproduce:
1.
2.
3.

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Sylvain_Juge · August 24, 2022, 1:20pm

Hi @Pratiksha_Nagoshe , welcome to the forum !

Could you elaborate a bit on the setup steps that you followed to add the agent layer and lambda extension that are both required on top of the environment variables ?

Do you see anything in the lambda standard output when it executes ? If the agent is properly added then you should have a few lines created by the Java agent.

Pratiksha_Nagoshe · August 24, 2022, 5:03pm

Hello @Sylvain_Juge
Added one more env variable : ELASTIC_APM_LOG_LEVEL TRACE
I followed this page Monitoring AWS Lambda Java Functions | APM Java Agent Reference [1.x] | Elastic for setup

Below are the logs
2022-08-24 16:55:54,749 [main] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - endTransaction := endTransaction is method to end transaction

at co.elastic.apm.agent.impl.ElasticApmTracer.startRootTransaction(ElasticApmTracer.java:171) [elastic-apm-agent.jar:1.33.0]
2022-08-24 16:55:54,748 [main] TRACE co.elastic.apm.agent.impl.ElasticApmTracer - starting transaction at

alexander.wert · August 25, 2022, 7:40am

Hi @Pratiksha_Nagoshe ,

could you please try with the newest release of the Elastic APM Lambda extension (v 1.1.0): Release v1.1.0 · elastic/apm-aws-lambda · GitHub

The new version of the extension has an improved logging in case of connection errors to the APM server.

Can you also provide the extension logs (which are also logged into the same CloudWatch streams) in DEBUG mode (not only the APM agent logs) so we can see what happens there.

Thanks

Pratiksha_Nagoshe · August 25, 2022, 8:21am

Hi @alexander.wert , Thanks for reply
I updated version of elastic-apm-extension-ver-1-1-0-x86_64

Please check below logs

2022-08-25 07:59:05,215 [elastic-apm-server-healthcheck] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Type match for instrumentation CreateSpanInstrumentation: (hasSuperType(erasure(is(class java.net.HttpURLConnection))) and not(name(equals(sun.net.www.protocol.https.HttpsURLConnectionImpl)))) matches class sun.net.www.protocol.http.HttpURLConnection
2022-08-25 07:59:05,215 [elastic-apm-server-healthcheck] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Type match for instrumentation CreateSpanInstrumentation: (hasSuperType(erasure(is(class java.net.HttpURLConnection))) and not(name(equals(sun.net.www.protocol.https.HttpsURLConnectionImpl)))) matches class sun.net.www.protocol.http.HttpURLConnection

2022-08-25 07:58:55,537 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Type match for instrumentation ExecutorRunnableInstrumentation: (hasSuperType(erasure(name(equals(java.util.concurrent.Executor)))) and not(name(equals(org.apache.felix.resolver.ResolverImpl$DumbExecutor))) and not(name(contains(jetty))) and not(name(contains(tomcat))) and not(name(contains(jboss))) and not(name(contains(undertow))) and not(name(contains(netty))) and not(name(contains(vertx))) and not(name(startsWith(com.hazelcast))) and not((name(contains($Proxy)) or name(contains($$)) or name(contains($JaxbAccessor)) or name(contains(CGLIB)) or name(contains(EnhancerBy))))) matches class java.util.concurrent.AbstractExecutorService
2022-08-25 07:58:55,537 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Type match for instrumentation ExecutorRunnableInstrumentation: (hasSuperType(erasure(name(equals(java.util.concurrent.Executor)))) and not(name(equals(org.apache.felix.resolver.ResolverImpl$DumbExecutor))) and not(name(contains(jetty))) and not(name(contains(tomcat))) and not(name(contains(jboss))) and not(name(contains(undertow))) and not(name(contains(netty))) and not(name(contains(vertx))) and not(name(startsWith(com.hazelcast))) and not((name(contains($Proxy)) or name(contains($$)) or name(contains($JaxbAccessor)) or name(contains(CGLIB)) or name(contains(EnhancerBy))))) matches class java.util.concurrent.AbstractExecutorService

2022-08-25 07:58:35,016 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Not applying excluded instrumentation co.elastic.apm.agent.vertx.v4.web.WebInstrumentation$ContextImplTracerInstrumentation

2022-08-25 07:59:37,018 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Method match for instrumentation ExecutePreparedStatementInstrumentation: ((name(equals(execute)) or name(equals(executeQuery))) and hasParameter(ofSize(0)) and isPublic() and not(isAbstract())) matches public synchronized java.sql.ResultSet com.vertica.jdbc.common.SCallableStatement.executeQuery() throws java.sql.SQLException

LOGS Name: apm-lambda-extension State: Subscribed Types: [platform]

{
"log.level": "warn",
"@timestamp": "2022-08-25T07:58:27.639Z",
"log.origin": {
"file.name": "extension/process_env.go",
"file.line": 110
},
"message": "Could not read ELASTIC_APM_DATA_RECEIVER_TIMEOUT_SECONDS, defaulting to 15: strconv.Atoi: parsing "": invalid syntax",
"ecs.version": "1.6.0"
}

{
"log.level": "warn",
"@timestamp": "2022-08-25T07:58:27.656Z",
"log.origin": {
"file.name": "extension/process_env.go",
"file.line": 116
},
"message": "Could not read ELASTIC_APM_DATA_FORWARDER_TIMEOUT_SECONDS, defaulting to 3: strconv.Atoi: parsing "": invalid syntax",
"ecs.version": "1.6.0"
}

{
"log.level": "debug",
"@timestamp": "2022-08-25T07:59:23.576Z",
"log.origin": {
"file.name": "logsapi/event.go",
"file.line": 78
},
"message": "Received log event platform.extension",
"ecs.version": "1.6.0"
}

alexander.wert · August 25, 2022, 12:03pm

Hi @Pratiksha_Nagoshe ,

is this the full logs?
I would expect something like this to be logged by the extension:

Or some error log stating that connection to APM server was not successful.

Pratiksha_Nagoshe · August 25, 2022, 4:33pm

Hey @alexander.wert
No, I am not getting such logs

Pratiksha_Nagoshe · September 2, 2022, 3:11am

Hi @alexander.wert and @Sylvain_Juge ,
Is there any configuration missing ?
please help with this
is properties file & jar file is required in upload ?
is code has to be compiled by specific java version ?
Thanks

alexander.wert · September 2, 2022, 2:15pm

Hi @Pratiksha_Nagoshe,

without more information about your setup it's hard to do further troubleshooting.
Can you please confirm that you have done all of the following:

You added BOTH layers to your function ( arn:aws:lambda:us-east-1:267093732750:layer:elastic-apm-extension-ver-1-1-0-x86_64:1 AND arn:aws:lambda:us-east-1:267093732750:layer:elastic-apm-java-ver-1-33-0:1) of course with the correct AWS region being replaced in the ARNs
You configured ALL of the following environment variables:

AWS_LAMBDA_EXEC_WRAPPER = /opt/elastic-apm-handler
ELASTIC_APM_LAMBDA_APM_SERVER =
ELASTIC_APM_SECRET_TOKEN = <YOUR_APM_SECRET_TOKEN>

In your description above I see that the environment variable AWS_LAMBDA_EXEC_WRAPPER is missing. Is it actually missing or did you just forgot to mention it in the description above?

Setting this variable is necessary for the agent to attach to the target application.
Could you maybe provide a screenshot (with obfuscated values) of your environment variables configuration and your layer configurations from the AWS Lambda Web console?

Thanks!

Pratiksha_Nagoshe · September 2, 2022, 2:40pm

@alexander.wert Thanks for reply
we added all configuration
arn:aws:lambda:us-east-1:267093732750:layer:elastic-apm-extension-ver-1-1-0-x86_64:1
arn:aws:lambda:us-east-1:267093732750:layer:elastic-apm-java-ver-1-33-0:1

we not added token because we remove that from our server

I hope this will helpful for you

In jar file , we added elastic.properties file and jar as well , is this effecting ?
is code has to be compiled by specific java version ?
Java 8 on Amazon Linux 1

Waiting for your reply
Thanks

alexander.wert · September 5, 2022, 6:17am

Hi @Pratiksha_Nagoshe,

I now see what's the problem here: You are running on the Java 8 Amazon Linux 1 Lambda runtime. As documented here, this runtime is not supported.

Can you try with Java 8 on Amazon Linux 2 OR Java 11 on Amazon Linux 2 and see if you still see this problem?

In jar file , we added elastic.properties file and jar as well , is this effecting ?
is code has to be compiled by specific java version ?

There's no need to add anything to your target JAR, the agent layer will provide everything that is needed. Regarding the Java compile version, the only requirement is that it needs to be Java 8 or higher.

Pratiksha_Nagoshe · September 5, 2022, 7:58am

Hi @alexander.wert ,

now complied with java 11 but still error

{
"log.level": "error",
"@timestamp": "2022-09-05T07:54:04.083Z",
"log.origin": {
"file.name": "app/run.go",
"file.line": 178
},
"message": "error sending to APM server, skipping: failed to post to APM server: Post "http://host:port/intake/v2/events": context deadline exceeded (Client.Timeout exceeded while awaiting headers)",
"ecs.version": "1.6.0",
"log.origin.stack_trace": "github.com/elastic/apm-aws-lambda/app.(*App).processEvent.func1\n\t/var/lib/jenkins/workspace/ibrary_apm-aws-lambda-mbp_v1.1.0/src/github.com/elastic/apm-aws-lambda/app/run.go:178"
}

Waiting for your reply
Thanks

simitt · September 5, 2022, 11:58am

Hi @Pratiksha_Nagoshe ,
the last message that you shared (error sending to APM server, skipping: failed to post to APM server: Post "http://host:port/intake/v2/events": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) might also indicate an issue with your APM Server. Can you check the logs in the APM Server to see if it is running into any issues and/or giving us more information?

axw · September 5, 2022, 12:21pm

@Pratiksha_Nagoshe one other thing you can check: is your APM Server even accessible? Can you use curl to check that the server is responding? It may be your APM Server host is firewalled.

In your initial post you wrote:

|ELASTIC_APM_LAMBDA_APM_SERVER|http://14.142.143.138:8200|

I can ping that IP, but port 8200 is not open.

Pratiksha_Nagoshe · September 7, 2022, 7:45am

Hi @alexander.wert , @Sylvain_Juge , @simitt , @axw Thanks reply
Its working with java 11 dummy aws lambda but with real application its showing following error logs

2022-09-07T12:42:27.166+0530 ERROR [request] middleware/log_middleware.go:74 data validation error {"request_id": "4dde2c43-770f-4ae1-a1d7-646fbf6d7f93", "method": "POST", "URL": "/intake/v2/events", "content_length": 405,

-2022-09-07 07:12:26,876 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type TRANSACTION with this error: null

"message": "APM server response body: {"accepted":0,"errors":[{"message":"did not recognize object type","document":"{\"metricset\":{\"samples\":{\"faas.coldstart_duration\":{\"value\":455.7799987792969},\"faas.timeout\":{\"value\":195000},\"system.memory.total\":{\"value\":5.36870912e+08},\"system.memory.actual.free\":{\"value\":4.15236096e+08},\"faas.duration\":{\"value\":14438.7099609375},\"faas.billed_duration\":{\"value\":14439}},\"timestamp\":1662545939270000,\"faas\":{\"coldstart\":true,\"execution\":\"11a9f891-19ed-4f26-90a9-74ed24b226ce\",\"id\":==========}}}}

2022-09-07T12:42:27.166+0530 ERROR [request] middleware/log_middleware.go:74 data validation error {"request_id": "4d-770f-4ae1-a1d7-646fbf6d7f93", "method": "POST", "URL": "/intake/v2/events", "content_length": 405, "remote_address": "*****", "user-agent": "Go-http-client/1.1", "response_code": 400, "error": "did not recognize object type [{"metricset":{"samples":{"faas.duration":{"value":65826.90625},"faas.billed_duration":{"value":65827},"faas.coldstart_duration":{"value":0},"faas.timeout":{"value":267000},"system.memory.total":{"value":5.36870912e+08},"system.memory.actual.free":{"value":3.60710144e+08}},"timestamp":1662534583442000,"faas":{"coldstart":false,"execution":"5a0f5171-f854-4bbb-bc8b-fb0","id":""}}}]"}

Thanks
waiting for reply

simitt · September 7, 2022, 11:14am

When setting the log level in the apm-server to debug (logging.level: "debug"), the server should log the full event body. Please share these log details here (but remove any sensitive information beforehand). It should give us more insights into the data validation error.

Pratiksha_Nagoshe · September 7, 2022, 11:58am

Hi @simitt Thanks for reply

please check the logs above I added full logs

axw · September 8, 2022, 3:03am

@Pratiksha_Nagoshe the APM Server log message does not line up with v7.15.0. Just guessing, but maybe you're actually running v7.5.0? That version does not have support for metrics, and I think lines up with the log message you have supplied.

Can you please check which version you're running, and provide the complete apm-server.log, with any sensitive data redacted?

Pratiksha_Nagoshe · September 9, 2022, 7:04am

Thanks all for such nice response
We changed java version to 11
It worked

Pratiksha_Nagoshe · September 12, 2022, 1:34pm

Hi @alexander.wert , @axw , @Sylvain_Juge , @simitt

for other lambda we got the error logs following
WARN co.elastic.apm.agent.impl.transaction.AbstractSpan - End has already been

2022-09-12 12:52:59,335 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type TRANSACTION with this error: null

2022-09-12 12:42:19,172 [main] DEBUG co.elastic.apm.agent.bci.bytebuddy.CustomElementMatchers - Implementation version parsing error: ProtectionDomain (file:/var/task/ )