"Service Unavailable" crashing the container

Elastic Observability APM
1

Kibana version : 7.6
Elasticsearch version : 7.6
APM Server version : 7.6
APM Agent language and version : Node.JS 3.3.0
Original install method : Elastic Cloud

Other:

  • Code running on Google Cloud Run (Knative container based serverless offering)
  • We have noticed this issue arrise when our Elastic cluster is under particularly high load

Sporadically we receive the following error in our logs which ends with exiting the process i.e. crashing the container. I would expect the below to not crash the container so it can carry on handling requests / doesn't cause a request to stop:

APM Server transport error (503): Unexpected APM Server response when polling config

{
  error: "Service Unavailable"   
  message: "Service Unavailable"   
  statusCode: 503   
 }

Unhandled rejection

ResponseError: Response Error at IncomingMessage.<anonymous> (/usr/src/app/node_modules/@elastic/elasticsearch/lib/Transport.js:294:25) at IncomingMessage.emit (events.js:208:15) at IncomingMessage.EventEmitter.emit (domain.js:471:20) at endReadableNT (_stream_readable.js:1154:12) at processTicksAndRejections (internal/process/task_queues.js:77:11)

Error: Process exited with code 16 at process.<anonymous> (/usr/src/app/node_modules/@google-cloud/functions-framework/build/src/invoker.js:393:29) at process.emit (events.js:203:13) at process.EventEmitter.emit (domain.js:471:20) at process.exit (internal/process/per_thread.js:158:15) at logAndSendError (/usr/src/app/node_modules/@google-cloud/functions-framework/build/src/invoker.js:184:9) at process.<anonymous> (/usr/src/app/node_modules/@google-cloud/functions-framework/build/src/invoker.js:390:13) at process.emit (events.js:203:13) at process.EventEmitter.emit (domain.js:471:20) at processPromiseRejections (internal/process/promises.js:201:33) at processTicksAndRejections (internal/process/task_queues.js:86:32)

Container called exit(16).

Note RE the code 16 - this is a fallback code the NPM module @google-cloud/functions-framework uses when there is an unhandled rejection

Is there a quick fix we can add to avoid this ending by crashing the container?

2

Potentially the above issue isn't related to the Elastic APM NPM module. I don't believe the Elastic APM NPM module uses the @elastic/elasticsearch NPM module at all (or any dependency of it)?

We do have another NPM module that uses @elastic/elasticsearch - am currently looking to see if this caused the issue

3

Thanks Mat,
any chance you can set the agent on debug and see if the problem reproduce ? (Troubleshooting | APM Node.js Agent Reference [4.x] | Elastic)
In any case , the app shouldn't crash...

4

This ended up being an internal issue - apologies for the thread! Thanks for the prompt reply though.

5

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.