Elastic APM: Instance rejected connection

[Anandakumar (Palanisamy)]

Kibana version: v8.7.1

Elasticsearch version: v8.7.1

APM Server version: v8.7.1

APM Agent language and version: Node.Js and using elastic-apm-node@4.0.0

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Hi,

I have a functioning APM server to which we are sending data from over 8 Node JS-based microservices. (using elastic-apm-node library).

Now, around yesterday I added a new microservice to our application and set up APM for the same using elastic-apm-node@4.0.0. And, it doesn't work.

Getting the following error in the logs.

{"log.level":"error","@timestamp":"2023-10-04T08:57:36.210Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (503): Unexpected APM Server response when polling config\nagentcfg kibana request failed with status code 502: {\"ok\":false,\"message\":\"The instance rejected the connection.\"}\n"}

Could you please advice what should be the next steps in resolving this?

[David_Luna_Bistuer]

Hi @Anandakumar,

we would need to check what configuration you have in the new added microservice. Since version 3.7 of the agent the 1st log to appear shows it (redacting sensitive information like tokens). Could you please share it?
Also it would be helpful to see how it is instrumented. Is a new service or an instance of an existing one?

Regards,
David

[Anandakumar (Palanisamy)]

Hi David, Looks like importing the apm.ts as the first line in the entry point file of the Nodejs service had solved the problem.

src/apm.ts:

import apm from 'elastic-apm-node';
import env from './env';

const isActive = ['production', 'development'].includes(env.NODE_ENV);

export default apm.start({
  active: isActive,
  environment: env.NODE_ENV,
});

apm.ts is imported into the entry point file server.ts; (Didn't work)

import app from './app';
import './utils/apm';

const port = process.env.PORT;

const server = app.listen(port, () => {
  logger.info({ port }, 'server listening');
  logger.info('Environment : ', process.env.NODE_ENV);
});

export default server;

The one that worked was this;

import './utils/apm';
import app from './app';

const port = process.env.PORT;

const server = app.listen(port, () => {
  logger.info({ port }, 'server listening');
  logger.info('Environment : ', process.env.NODE_ENV);
});

export default server;

The documentation here is asking that line to be at the top in the first file loaded in the app. But, it's indicated as a comment in the example code.

Would be useful, if that's properly included as a pointer in the documentation. Right now, its very easy to miss.

thanks, Anand.