Unable to reach APM Server: 'Connection aborted.' RemoteDisconnected('Remote end closed connection without response')) (url: http://127.0.0.1:8200/intake/v2/events). (potentially caused by apm-lambda-extension/extension.handleIntakeV2Events exception)

Anton_Matviyenko
Mon, 13 Nov 2023 16:49:50 +0000
APM

Dear Elastic Team,

It looks like this issue is similar to the following:

Failed to submit message (aws lambda using python) APM

Hello guys, I am using elastic in lambda, with: elastic-apm==6.14 layer:elastic-apm-extension-ver-1-3-0-x86_64:1 python 3.9 in aws lambda I am receiving an error in some operations, most of the executions works fine, but i have some erros like this: Failed to submit message: "Unable to reach APM Server: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) Did someone resolved this error ? Thanks guys

Unable to reach APM Server: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',)) APM

Kibana version: 6.6.0 Elasticsearch version: 6.6.0 APM Server version: 6.6.0 APM Agent language and version: Python Browser version: FF 65.0 Original install method (e.g. download page, yum, deb, from source, etc.) and version: apt Fresh install or upgraded from other version? Fresh install I got this issue. Does anybody know how to fix it? Failed to submit message: u"Unable to reach APM Server: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))…

We do use Python APM Elastic Agent along with Sentry and in Sentry we do see the following exceptions:

Failed to submit message: "Unable to reach APM Server: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) (url: http://127.0.0.1:8200/intake/v2/events)

Elasticsearch version:: v8.10.2

APM Server version: v8.10.2

APM Agent language and version: 6.19.0

Original install method (e.g. download page, yum, deb, from source, etc.) and version: with:

1. template.yaml Layers: arn:aws:lambda:eu-central-1:267093732750:layer:elastic-apm-extension-ver-1-0-2-x86_64:1

2. pip requirements.txt: elastic-apm==6.19.0

I enabled the ELASTIC_APM_LOG_LEVEL: debug in lambda "template.yaml" and captured the logs for the failed request.

Please note that elastic apm python module is installed as layer and as a python module during lambda build (hopefully that is fine).

Right now about the behaviour - it is working fine there is no misconfigurations. The apm transaction information is being properly shipped. From time to time this exception is happening (which is related to the golang library crash - see the log). If such crash is experienced then this transaction information is not being shipped properly.

Provide logs and/or server output (if relevant):

[DEBUG]	2023-11-13T15:55:10.826Z	some-guid-here	forced flush
{
    "log.level": "debug",
    "@timestamp": "2023-11-13T15:55:10.827Z",
    "log.origin": {
        "file.name": "extension/route_handlers.go",
        "file.line": 78
    },
    "message": "Handling APM Data Intake",
    "ecs.version": "1.6.0"
}

{
    "log.level": "debug",
    "@timestamp": "2023-11-13T15:55:10.827Z",
    "log.origin": {
        "file.name": "extension/apm_server_transport.go",
        "file.line": 238
    },
    "message": "Adding agent data to buffer to be sent to apm server",
    "ecs.version": "1.6.0"
}

2023/11/13 15:55:10 http: panic serving 127.0.0.1:43610: send on closed channel
goroutine 86 [running]:
net/http.(*conn).serve.func1()
/var/lib/jenkins/workspace/ibrary_apm-aws-lambda-mbp_v1.0.2/.gvm/versions/go1.18.2.linux.amd64/src/net/http/server.go:1825 +0xbf
panic({0x6d3840, 0x7a5990})
/var/lib/jenkins/workspace/ibrary_apm-aws-lambda-mbp_v1.0.2/.gvm/versions/go1.18.2.linux.amd64/src/runtime/panic.go:844 +0x258
elastic/apm-lambda-extension/extension.handleIntakeV2Events.func1({0x7a8b38, 0xc0001b4ee0}, 0xc000287200)
/var/lib/jenkins/workspace/ibrary_apm-aws-lambda-mbp_v1.0.2/src/github.com/elastic/apm-aws-lambda/apm-lambda-extension/extension/route_handlers.go:97 +0x497
net/http.HandlerFunc.ServeHTTP(0x70f27ac438?, {0x7a8b38?, 0xc0001b4ee0?}, 0xc0002938a0?)
/var/lib/jenkins/workspace/ibrary_apm-aws-lambda-mbp_v1.0.2/.gvm/versions/go1.18.2.linux.amd64/src/net/http/server.go:2084 +0x2f
net/http.(*ServeMux).ServeHTTP(0xc0001db097?, {0x7a8b38, 0xc0001b4ee0}, 0xc000287200)
/var/lib/jenkins/workspace/ibrary_apm-aws-lambda-mbp_v1.0.2/.gvm/versions/go1.18.2.linux.amd64/src/net/http/server.go:2462 +0x149
net/http.serverHandler.ServeHTTP({0x7a7d20?}, {0x7a8b38, 0xc0001b4ee0}, 0xc000287200)
/var/lib/jenkins/workspace/ibrary_apm-aws-lambda-mbp_v1.0.2/.gvm/versions/go1.18.2.linux.amd64/src/net/http/server.go:2916 +0x43b
net/http.(*conn).serve(0xc00018c000, {0x7a8d90, 0xc000074600})
/var/lib/jenkins/workspace/ibrary_apm-aws-lambda-mbp_v1.0.2/.gvm/versions/go1.18.2.linux.amd64/src/net/http/server.go:1966 +0x5d7
created by net/http.(*Server).Serve
/var/lib/jenkins/workspace/ibrary_apm-aws-lambda-mbp_v1.0.2/.gvm/versions/go1.18.2.linux.amd64/src/net/http/server.go:3071 +0x4db
{
    "log.level": "debug",
    "@timestamp": "2023-11-13T15:55:10.827Z",
    "log.origin": {
        "file.name": "apm-lambda-extension/main.go",
        "file.line": 125
    },
    "message": "Received event.",
    "ecs.version": "1.6.0"
}

Thank you for your Support!

3 posts - 3 participants