Node Application crashes when apm-server is not online

[ashuSvirus (Ashutosh Mittal)]

I have also increased the ulimit open-files to 1000000.

Kibana version: 6.5.4

Elasticsearch version: 6.5.4

APM Server version: 6.5.4

APM Agent language and version: Nodejs and version 2.6.0 elastic-apm-node

Browser version:

Original install method (e.g. download page, yum, deb, from source, etc.) and version: deb

Fresh install or upgraded from other version? Fresh

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Steps to reproduce:

1. Shut down APM-server
2. Send 1000 req/s

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

APM Server transport error (ENOTFOUND): getaddrinfo ENOTFOUND localhost localhost:8200
APM Server transport error (EMFILE): getaddrinfo EMFILE localhost:8200
APM Server transport error (EMFILE): getaddrinfo EMFILE localhost:8200

[wa7son (Thomas Watson)]

Thanks for reporting this issue. I've been trying to reproduce this on MacOS, but so far without any luck. If this is file descriptor related, it could very well be OS dependent of course.

I'm a little unsure if this is related to span serialization or the actual TCP socket creation. But if it's related to the latter, this would probably go away if we added a simple graceful backoff algorithm when ever connection issues occurred. I've created an issue to get this implemented: https://github.com/elastic/apm-nodejs-http-client/issues/47