Hello guys,
I am using elastic in lambda, with:
I am receiving an error in some operations, most of the executions works fine, but i have some erros like this:
Failed to submit message: "Unable to reach APM Server: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
Did someone resolved this error ?
Thanks guys
Can you show your (redacted) environment variables?
@lahsivjar have you seen anything like this from the extension?
Sure. I am using SAM, the template.yml contains the environment variables:
ELASTIC_APM_SERVICE_NAME: "service-name"
ELASTIC_APM_SEND_STRATEGY: background
ELASTIC_APM_SECRET_TOKEN:
Ref: ElasticAPMSecretToken
ELASTIC_APM_LAMBDA_APM_SERVER: http://elastic.prd.mybusiness.net:8200
ELASTIC_APM_ENVIRONMENT: PRD
Layer:
Layers:
- arn:aws:lambda:XXXXXXX:layer:elastic-apm-extension-ver-1-3-0-x86_64:1
Lambda execution works fine, all errors are handled by elastic, but in this cases we have the error traceback logging in another tool that we use for logs.
Complete traceback:
[ERROR] (redacted) ailed to submit message: "Unable to reach APM Server: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) (url: http://localhost:8200/intake/v2/events)"
Traceback (most recent call last):
File "/opt/python/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/opt/python/urllib3/connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/opt/python/urllib3/connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
File "/opt/python/sentry_sdk/integrations/stdlib.py", line 121, in getresponse
rv = real_getresponse(self, *args, **kwargs)
File "/var/lang/lib/python3.8/http/client.py", line 1348, in getresponse
response.begin()
File "/var/lang/lib/python3.8/http/client.py", line 316, in begin
version, status, reason = self._read_status()
File "/var/lang/lib/python3.8/http/client.py", line 285, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/python/elasticapm/transport/http.py", line 90, in send
response = self.http.urlopen(
File "/opt/python/urllib3/poolmanager.py", line 376, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/opt/python/elasticapm/instrumentation/packages/base.py", line 206, in call_if_sampling
return wrapped(*args, **kwargs)
File "/opt/python/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/opt/python/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/opt/python/urllib3/packages/six.py", line 769, in reraise
raise value.with_traceback(tb)
File "/opt/python/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/opt/python/urllib3/connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/opt/python/urllib3/connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
File "/opt/python/sentry_sdk/integrations/stdlib.py", line 121, in getresponse
rv = real_getresponse(self, *args, **kwargs)
File "/var/lang/lib/python3.8/http/client.py", line 1348, in getresponse
response.begin()
File "/var/lang/lib/python3.8/http/client.py", line 316, in begin
version, status, reason = self._read_status()
File "/var/lang/lib/python3.8/http/client.py", line 285, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/python/elasticapm/transport/base.py", line 279, in _flush
self.send(data, forced_flush=forced_flush)
File "/opt/python/elasticapm/transport/http.py", line 104, in send
raise TransportException(message, data, print_trace=print_trace)
elasticapm.transport.exceptions.TransportException: Unable to reach APM Server: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) (url: http://localhost:8200/intake/v2/events)
[ERROR] 2023-05-03T19:45:26.858Z 4cd917fd-3fb3-5206-97c1-4b31843f4bce Failed to submit message: "Unable to reach APM Server: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) (url: http://localhost:8200/intake/v2/events)" Traceback (most recent call last): File "/opt/python/urllib3/connectionpool.py", line 703, in urlopen httplib_response = self._make_request( File "/opt/python/urllib3/connectionpool.py", line 449, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "/opt/python/urllib3/connectionpool.py", line 444, in _make_request httplib_response = conn.getresponse() File "/opt/python/sentry_sdk/integrations/stdlib.py", line 121, in getresponse rv = real_getresponse(self, *args, **kwargs) File "/var/lang/lib/python3.8/http/client.py", line 1348, in getresponse response.begin() File "/var/lang/lib/python3.8/http/client.py", line 316, in begin version, status, reason = self._read_status() File "/var/lang/lib/python3.8/http/client.py", line 285, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/python/elasticapm/transport/http.py", line 90, in send response = self.http.urlopen( File "/opt/python/urllib3/poolmanager.py", line 376, in urlopen response = conn.urlopen(method, u.request_uri, **kw) File "/opt/python/elasticapm/instrumentation/packages/base.py", line 206, in call_if_sampling return wrapped(*args, **kwargs) File "/opt/python/urllib3/connectionpool.py", line 787, in urlopen retries = retries.increment( File "/opt/python/urllib3/util/retry.py", line 550, in increment raise six.reraise(type(error), error, _stacktrace) File "/opt/python/urllib3/packages/six.py", line 769, in reraise raise value.with_traceback(tb) File "/opt/python/urllib3/connectionpool.py", line 703, in urlopen httplib_response = self._make_request( File "/opt/python/urllib3/connectionpool.py", line 449, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "/opt/python/urllib3/connectionpool.py", line 444, in _make_request httplib_response = conn.getresponse() File "/opt/python/sentry_sdk/integrations/stdlib.py", line 121, in getresponse rv = real_getresponse(self, *args, **kwargs) File "/var/lang/lib/python3.8/http/client.py", line 1348, in getresponse response.begin() File "/var/lang/lib/python3.8/http/client.py", line 316, in begin version, status, reason = self._read_status() File "/var/lang/lib/python3.8/http/client.py", line 285, in _read_status raise RemoteDisconnected("Remote end closed connection without" urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/python/elasticapm/transport/base.py", line 279, in _flush self.send(data, forced_flush=forced_flush) File "/opt/python/elasticapm/transport/http.py", line 104, in send raise TransportException(message, data, print_trace=print_trace) elasticapm.transport.exceptions.TransportException: Unable to reach APM Server: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) (url: http://localhost:8200/intake/v2/events)
Thanks @basepi
No, haven't seen such errors before, the extension seems to be aborting the connection from the python agent.
@gabriel-f-santos Will you be able to share the extension logs in debug mode when the issue happens? Debug mode can be enabled by setting the environment variable ELASTIC_APM_LOG_LEVEL to debug.
You can get the extension logs from Cloudwatch.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.