APM Node Agent: Exception is captured by APM, but is not being sent to the APM Server

[alexpereira (Alexsander Pereira)]

We added the APM Node in our code inside the file server.js, being the first thing to load. However if there is an exception within the server.js file APM will capture the same but will not send it to the server.

The following log shows the catch of the exception. But we do not see her in Kibana. Other exceptions we can see normally.

Log debug:

shimming http@6.14.4 module
shimming http.Server.prototype.emit function
shimming http.request function
shimming http.ServerResponse.prototype.writeHead function
shimming https@6.14.4 module
shimming https.Server.prototype.emit function
shimming finalhandler@1.1.1 module
shimming express@4.16.4 module
shimming express.Router.process_params function
shimming express.Router.use function
shimming express.static function
copying property mime from express.static
shimming bluebird@3.5.3 module
shimming bluebird.prototype functions: [ '_then', '_addCallbacks' ]
shimming bluebird.config
skip shimming layer.handle_request (layer: , path: /)
Elastic APM caught unhandled exception: aaa is not defined
Sending error to Elastic APM { id: 'c58a14ce2494c0a7697a56cd2ebc3120' }
no active transaction found - cannot build new span
intercepted call to http.request %o { id: null }

Payload log:

{"metadata":{"service":{"name":"api-insights-development","runtime":{"name":"node","version":"6.14.4"},"language":{"name":"javascript"},"agent":{"name":"nodejs","version":"2.0.1"}},"process":{"pid":23264,"ppid":23254,"title":"node","argv":["/opt/elasticbeanstalk/node-install/node-v6.14.4-linux-x64/bin/node","/var/app/current"]},"system":{"hostname":"api-insights-development-ip-10-0-3-140","architecture":"x64","platform":"linux"}}}
{"error":{"exception":{"message":"aaa is not defined","type":"ReferenceError","stacktrace":[{"filename":"server/server.js","lineno":71,"function":"Object.","library_frame":false,"abs_path":"/var/app/current/server/server.js","pre_context":["};",""],"context_line":"aaa();","post_context":["","boot(app, bootOptions, function(err) {"]},{"filename":"module.js","lineno":577,"function":"Module._compile","library_frame":true,"abs_path":"module.js"},{"filename":"module.js","lineno":586,"function":"Module._extensions..js","library_frame":true,"abs_path":"module.js"},{"filename":"module.js","lineno":494,"function":"Module.load","library_frame":true,"abs_path":"module.js"},{"filename":"module.js","lineno":453,"function":"tryModuleLoad","library_frame":true,"abs_path":"module.js"},{"filename":"module.js","lineno":445,"function":"Module._load","library_frame":true,"abs_path":"module.js"},{"filename":"module.js","lineno":611,"function":"Module.runMain","library_frame":true,"abs_path":"module.js"},{"filename":"bootstrap_node.js","lineno":394,"function":"run","library_frame":true,"abs_path":"bootstrap_node.js"},{"filename":"bootstrap_node.js","lineno":160,"function":"startup","library_frame":true,"abs_path":"bootstrap_node.js"},{"filename":"bootstrap_node.js","lineno":507,"function":"","library_frame":true,"abs_path":"bootstrap_node.js"}],"handled":false},"culprit":"Object. (server/server.js)","id":"c58a14ce2494c0a7697a56cd2ebc3120","timestamp":1544636558941000,"context":{"user":{},"tags":{},"custom":{}}}}

[wa7son (Thomas Watson)]

The agent should normally hold open the process until the error event have been sent to the APM Server. However, if you have another module or code that hooks into the uncaughtException event and manually shut down your app, that might exit the process before the error has been successfully sent to the APM Server.

If that's not the case, it might be a race-condition where the internals of Node.js or the OS doesn't completely flush the socket before Node.js exits and then the socket is terminated prematurely. I haven't seen this my self, but can't rule it out completely.

You can see if this is the case bu adding this code:

agent.handleUncaughtException(function () {
  console.error('Done sending error to APM Server - exiting...')
  process.exit(1)
})

If the log line in the callback is logged to STDERR, but the error is still not sent to the APM Server, try to add a delay before the process.exit, eg:

agent.handleUncaughtException(function () {
  console.error('Done sending error to APM Server')
  setTimeout(function () {
    console.error('Exiting...')
    process.exit(1)
  }, 100)
})