Unable to reach APM Server: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

thienngho · January 31, 2019, 8:45am

Kibana version: 6.6.0

Elasticsearch version: 6.6.0

APM Server version: 6.6.0

APM Agent language and version: Python

Browser version: FF 65.0

Original install method (e.g. download page, yum, deb, from source, etc.) and version: apt

Fresh install or upgraded from other version? Fresh install

I got this issue. Does anybody know how to fix it?

Failed to submit message: u"Unable to reach APM Server: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',)) (url: http://10.10.10.10:8200/intake/v2/events)"

simitt · February 1, 2019, 12:33pm

Hi @thienngho,
which version of the agent are you using?

Have you changed any of the default configuration? Please also check your setup, e.g. ensure the APM Server is reachable by the agent, the proper APM Server host and port are defined, etc. Do you use any additional proxies between the agent and server communication?

thienngho · February 1, 2019, 1:33pm

Hi @simitt,

I'm using Python agent 4.1.0.
I didn't change any of the default configuration.
I tested the connectivity, all good.
I don't use any proxy.

It's not happened all the time, sometimes it happens

beniwohli · February 1, 2019, 2:14pm

@thienngho do you see any log messages from the APM Server at the same time as you get this error from your Python app? You might have to set up logging first though.

Piotr_Jankowski · February 4, 2019, 4:07pm

I have the same issue with elastic-apm python agent under flask. Downgrading to 4.0.3 makes this issue disappear. Furthermore when using 4.1 with apm-server 6.5 caused messages not to appear on the backend. Upgrading to 6.6.0 fixed it but tracebacks like above were still showing up in the flask logs. Downgrading to apm python agent to 4.0.3 with apm-server 6.6.0 work correctly.

This seems to be an issue with elastic-apm python agent.

beniwohli · February 5, 2019, 8:02am

@Piotr_Jankowski thanks for confirming the issue! How frequently did this issue happen (only every so often, or with every request from the Python agent to the APM Server)? And did you see anything in the logs from the APM Server at the same time you saw these tracebacks from the Python agent? And lastly, which Python version and urllib3 version do you use?

vlinhart · February 6, 2019, 11:12am

Having the same errors. I'd say its 10% with errors. Hard to say. My stack is on 6.5. apm python agent 4.0.3 has way less errors like this than the 4.1. However even apm python agent 4.0.3 does it in like 1% of situations. The apm logs have nothing - only logging success. Python 3.6.5.
urllib3==1.24.1 # via botocore, django-revproxy, elastic-apm, requests

Traceback (most recent call last):
  File "/home/deploy/my_proj/live/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/home/deploy/my_proj/live/lib/python3.6/site-packages/newrelic/hooks/external_urllib3.py", line 16, in _nr_wrapper_make_request_
    return wrapped(*args, **kwargs)
  File "/home/deploy/my_proj/live/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/home/deploy/my_proj/live/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/home/deploy/my_proj/live/lib/python3.6/site-packages/raven/breadcrumbs.py", line 390, in getresponse
    rv = real_getresponse(self, *args, **kwargs)
  File "/home/deploy/my_proj/live/lib/python3.6/site-packages/newrelic/hooks/external_httplib.py", line 69, in httplib_getresponse_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/usr/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.6/http/client.py", line 266, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/deploy/my_proj/live/lib/python3.6/site-packages/elasticapm/transport/http.py", line 40, in send
    "POST", url, body=data, headers=self._headers, timeout=self._timeout, preload_content=False
  File "/home/deploy/my_proj/live/lib/python3.6/site-packages/urllib3/poolmanager.py", line 323, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/home/deploy/my_proj/live/lib/python3.6/site-packages/elasticapm/instrumentation/packages/base.py", line 106, in call_if_sampling
    return wrapped(*args, **kwargs)
  File "/home/deploy/my_proj/live/lib/python3.6/site-packages/elasticapm/instrumentation/packages/base.py", line 106, in call_if_sampling
    return wrapped(*args, **kwargs)
  File "/home/deploy/my_proj/live/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/home/deploy/my_proj/live/lib/python3.6/site-packages/urllib3/util/retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/deploy/my_proj/live/lib/python3.6/site-packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/deploy/my_proj/live/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/home/deploy/my_proj/live/lib/python3.6/site-packages/newrelic/hooks/external_urllib3.py", line 16, in _nr_wrapper_make_request_
    return wrapped(*args, **kwargs)
  File "/home/deploy/my_proj/live/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/home/deploy/my_proj/live/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/home/deploy/my_proj/live/lib/python3.6/site-packages/raven/breadcrumbs.py", line 390, in getresponse
    rv = real_getresponse(self, *args, **kwargs)
  File "/home/deploy/my_proj/live/lib/python3.6/site-packages/newrelic/hooks/external_httplib.py", line 69, in httplib_getresponse_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/usr/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.6/http/client.py", line 266, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/deploy/my_proj/live/lib/python3.6/site-packages/elasticapm/transport/base.py", line 184, in send_sync
    self.handle_transport_fail(exception=e)
  File "/home/deploy/my_proj/live/lib/python3.6/site-packages/elasticapm/transport/http.py", line 53, in send
    raise TransportException(message, data, print_trace=print_trace)
elasticapm.transport.base.TransportException: Unable to reach APM Server: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',)) (url: http://es1.my_proj.com:8200/intake/v2/events)

beniwohli · February 6, 2019, 1:05pm

@vlinhart awesome, thanks for all the information!

One possibility why this is happening more often on 4.1 is due to the new metrics collection feature, as it regularly collects and sends data.

My current theory is that this could be due to a keep-alive connection that gets closed in-between requests. If that is the case, we could probably get away with simply retrying to send the payload when this happens. But I need to investigate this a bit more before going down that road.

vlinhart · February 6, 2019, 1:35pm

@beniwohli what does it regularly collects? I thought it's done only on requests. Though I noticed it was sending something in dev even when no requests were done.

Ok don't know how is the keep-alive used, I'd go without it or make it configurable? Thanks for looking into it!

vlinhart · February 6, 2019, 2:01pm

Seems it's the new
https://www.elastic.co/guide/en/apm/agent/python/current/metric-sets.html
How do I disable it, can't find it in the docs.
METRICS_INTERVAL=0ms ?

cjensen · February 10, 2019, 4:59am

I have apm-server, ES and Kibana on my local network. The app is running under a docker-compose environment.

Nothing in the APM server logs is showing an error - there is a log for the request itself and it does look successful but Python thinks it is not.

Here is an extended stacktrace:

web_1      | Failed to submit message: "Unable to reach APM Server: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) (url: http://192.168.16.100:8200/intake/v2/events)"
web_1      | Traceback (most recent call last):
web_1      |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
web_1      |     chunked=chunked)
web_1      |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request
web_1      |     six.raise_from(e, None)
web_1      |   File "<string>", line 2, in raise_from
web_1      |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request
web_1      |     httplib_response = conn.getresponse()
web_1      |   File "/usr/local/lib/python3.7/http/client.py", line 1321, in getresponse
web_1      |     response.begin()
web_1      |   File "/usr/local/lib/python3.7/http/client.py", line 296, in begin
web_1      |     version, status, reason = self._read_status()
web_1      |   File "/usr/local/lib/python3.7/http/client.py", line 265, in _read_status
web_1      |     raise RemoteDisconnected("Remote end closed connection without"
web_1      | http.client.RemoteDisconnected: Remote end closed connection without response
web_1      | 
web_1      | During handling of the above exception, another exception occurred:
web_1      | 
web_1      | Traceback (most recent call last):
web_1      |   File "/usr/local/lib/python3.7/site-packages/elasticapm/transport/http.py", line 40, in send
web_1      |     "POST", url, body=data, headers=self._headers, timeout=self._timeout, preload_content=False
web_1      |   File "/usr/local/lib/python3.7/site-packages/urllib3/poolmanager.py", line 323, in urlopen
web_1      |     response = conn.urlopen(method, u.request_uri, **kw)
web_1      |   File "/usr/local/lib/python3.7/site-packages/elasticapm/instrumentation/packages/base.py", line 106, in call_if_sampling
web_1      |     return wrapped(*args, **kwargs)
web_1      |   File "/usr/local/lib/python3.7/site-packages/elasticapm/instrumentation/packages/base.py", line 106, in call_if_sampling
web_1      |     return wrapped(*args, **kwargs)
web_1      |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
web_1      |     _stacktrace=sys.exc_info()[2])
web_1      |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 367, in increment
web_1      |     raise six.reraise(type(error), error, _stacktrace)
web_1      |   File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 685, in reraise
web_1      |     raise value.with_traceback(tb)
web_1      |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
web_1      |     chunked=chunked)
web_1      |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request
web_1      |     six.raise_from(e, None)
web_1      |   File "<string>", line 2, in raise_from
web_1      |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request
web_1      |     httplib_response = conn.getresponse()
web_1      |   File "/usr/local/lib/python3.7/http/client.py", line 1321, in getresponse
web_1      |     response.begin()
web_1      |   File "/usr/local/lib/python3.7/http/client.py", line 296, in begin
web_1      |     version, status, reason = self._read_status()
web_1      |   File "/usr/local/lib/python3.7/http/client.py", line 265, in _read_status
web_1      |     raise RemoteDisconnected("Remote end closed connection without"
web_1      | urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
web_1      | 
web_1      | During handling of the above exception, another exception occurred:
web_1      | 
web_1      | Traceback (most recent call last):
web_1      |   File "/usr/local/lib/python3.7/site-packages/elasticapm/transport/base.py", line 184, in send_sync
web_1      |     self.sync_transport.send(self, data)
web_1      |   File "/usr/local/lib/python3.7/site-packages/elasticapm/transport/http.py", line 53, in send
web_1      |     raise TransportException(message, data, print_trace=print_trace)
web_1      | elasticapm.transport.base.TransportException: Unable to reach APM Server: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) (url: http://192.168.16.100:8200/intake/v2/events)

Piotr_Jankowski · February 12, 2019, 3:39pm

I managed to make a reproduction script and run it through git bisect. I put the result in the github ticket: https://github.com/elastic/apm-agent-python/issues/409#issuecomment-462800909

system · March 5, 2019, 11:39am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.