Elastic APM server errors 502

lalitprasanth · August 14, 2023, 6:03am

Kibana version: 8.8.1

Elasticsearch version: 8.8.1

APM Server version: 8.8.1

APM Agent language and version: NodeJs Agent elastic-apm-node": "^3.47.0

Configuration:

const apm = require('elastic-apm-node').start({
  serviceName: 'name',
  secretToken: config.app.apmSecretToken,

  // Set the custom APM Server URL (default: http://localhost:8200)
  serverUrl: config.app.elasticApmUrl,
  ignoreUrls: ['/', '/metrics'],
  captureBody: 'transactions',
  logLevel: 'info',
  apiRequestSize: '1mb',
  environment: config.config_id
});

Using this configuration we were able to trace all our applications. But still we are seeing huge error log related to elastic-apm. Attaching the logs for your reference. This error logs are casing storage issue on our elastic cluster.

1691989228257	"{"log.level":"error","@timestamp":"2023-08-14T05:00:28.257Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691989288188	"{"log.level":"error","@timestamp":"2023-08-14T05:01:28.188Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": use of closed network connection
"}"
1691989301936	"{"log.level":"error","@timestamp":"2023-08-14T05:01:41.936Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (503): Unexpected APM Server response when polling config
{\"ok\":false,\"message\":\"The requested resource is currently unavailable.\"}
"}"
1691989307994	"{"log.level":"error","@timestamp":"2023-08-14T05:01:47.994Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
{\"ok\":false,\"message\":\"The instance rejected the connection.\"}
"}"
1691989318971	"{"log.level":"error","@timestamp":"2023-08-14T05:01:58.971Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
{\"ok\":false,\"message\":\"The instance rejected the connection.\"}
"}"
1691989348190	"{"log.level":"error","@timestamp":"2023-08-14T05:02:28.189Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691989378190	"{"log.level":"error","@timestamp":"2023-08-14T05:02:58.190Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691989438193	"{"log.level":"error","@timestamp":"2023-08-14T05:03:58.193Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691989468192	"{"log.level":"error","@timestamp":"2023-08-14T05:04:28.192Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691989528205	"{"log.level":"error","@timestamp":"2023-08-14T05:05:28.205Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691989558192	"{"log.level":"error","@timestamp":"2023-08-14T05:05:58.192Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691989622177	"{"log.level":"error","@timestamp":"2023-08-14T05:07:02.177Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (400): Unexpected APM Server response
APM Server accepted 0 events in the last request
Error: read tcp 172.17.0.14:8200->172.22.3.245:41096: i/o timeout"}"
1691989712249	"{"log.level":"error","@timestamp":"2023-08-14T05:08:32.249Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691989742264	"{"log.level":"error","@timestamp":"2023-08-14T05:09:02.264Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691989772276	"{"log.level":"error","@timestamp":"2023-08-14T05:09:32.276Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691989802296	"{"log.level":"error","@timestamp":"2023-08-14T05:10:02.295Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691989862359	"{"log.level":"error","@timestamp":"2023-08-14T05:11:02.359Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691989892367	"{"log.level":"error","@timestamp":"2023-08-14T05:11:32.367Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691989908450	"{"log.level":"error","@timestamp":"2023-08-14T05:11:48.450Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691989922397	"{"log.level":"error","@timestamp":"2023-08-14T05:12:02.397Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691989982440	"{"log.level":"error","@timestamp":"2023-08-14T05:13:02.439Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691990012464	"{"log.level":"error","@timestamp":"2023-08-14T05:13:32.463Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691990042485	"{"log.level":"error","@timestamp":"2023-08-14T05:14:02.484Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691990102528	"{"log.level":"error","@timestamp":"2023-08-14T05:15:02.528Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691990132549	"{"log.level":"error","@timestamp":"2023-08-14T05:15:32.548Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": use of closed network connection
"}"
1691990148647	"{"log.level":"error","@timestamp":"2023-08-14T05:15:48.647Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": use of closed network connection
"}"
1691990192593	"{"log.level":"error","@timestamp":"2023-08-14T05:16:32.593Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691990222594	"{"log.level":"error","@timestamp":"2023-08-14T05:17:02.594Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691990252625	"{"log.level":"error","@timestamp":"2023-08-14T05:17:32.625Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691990282654	"{"log.level":"error","@timestamp":"2023-08-14T05:18:02.654Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691990312668	"{"log.level":"error","@timestamp":"2023-08-14T05:18:32.667Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691990462769	"{"log.level":"error","@timestamp":"2023-08-14T05:21:02.768Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691990492783	"{"log.level":"error","@timestamp":"2023-08-14T05:21:32.783Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691990522810	"{"log.level":"error","@timestamp":"2023-08-14T05:22:02.810Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691990552836	"{"log.level":"error","@timestamp":"2023-08-14T05:22:32.836Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691990582867	"{"log.level":"error","@timestamp":"2023-08-14T05:23:02.867Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691990642927	"{"log.level":"error","@timestamp":"2023-08-14T05:24:02.926Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691990672957	"{"log.level":"error","@timestamp":"2023-08-14T05:24:32.957Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691990702986	"{"log.level":"error","@timestamp":"2023-08-14T05:25:02.986Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691990793075	"{"log.level":"error","@timestamp":"2023-08-14T05:26:33.074Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691990823105	"{"log.level":"error","@timestamp":"2023-08-14T05:27:03.105Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691990853135	"{"log.level":"error","@timestamp":"2023-08-14T05:27:33.135Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691990883166	"{"log.level":"error","@timestamp":"2023-08-14T05:28:03.166Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691990943203	"{"log.level":"error","@timestamp":"2023-08-14T05:29:03.203Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691990973206	"{"log.level":"error","@timestamp":"2023-08-14T05:29:33.206Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691991033258	"{"log.level":"error","@timestamp":"2023-08-14T05:30:33.258Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691991183397	"{"log.level":"error","@timestamp":"2023-08-14T05:33:03.397Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691991213425	"{"log.level":"error","@timestamp":"2023-08-14T05:33:33.425Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691991243430	"{"log.level":"error","@timestamp":"2023-08-14T05:34:03.430Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691991303484	"{"log.level":"error","@timestamp":"2023-08-14T05:35:03.484Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691991333517	"{"log.level":"error","@timestamp":"2023-08-14T05:35:33.517Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691991363535	"{"log.level":"error","@timestamp":"2023-08-14T05:36:03.535Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691991453613	"{"log.level":"error","@timestamp":"2023-08-14T05:37:33.613Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691991483646	"{"log.level":"error","@timestamp":"2023-08-14T05:38:03.646Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691991513666	"{"log.level":"error","@timestamp":"2023-08-14T05:38:33.666Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691991603744	"{"log.level":"error","@timestamp":"2023-08-14T05:40:03.743Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691991633777	"{"log.level":"error","@timestamp":"2023-08-14T05:40:33.777Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691991663787	"{"log.level":"error","@timestamp":"2023-08-14T05:41:03.787Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691991723844	"{"log.level":"error","@timestamp":"2023-08-14T05:42:03.844Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691991753873	"{"log.level":"error","@timestamp":"2023-08-14T05:42:33.873Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691991783915	"{"log.level":"error","@timestamp":"2023-08-14T05:43:03.915Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": use of closed network connection
"}"
1691991904015	"{"log.level":"error","@timestamp":"2023-08-14T05:45:04.015Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691991934046	"{"log.level":"error","@timestamp":"2023-08-14T05:45:34.046Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691991964075	"{"log.level":"error","@timestamp":"2023-08-14T05:46:04.075Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691991994105	"{"log.level":"error","@timestamp":"2023-08-14T05:46:34.105Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691992084185	"{"log.level":"error","@timestamp":"2023-08-14T05:48:04.184Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691992114185	"{"log.level":"error","@timestamp":"2023-08-14T05:48:34.185Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691992144199	"{"log.level":"error","@timestamp":"2023-08-14T05:49:04.199Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691992174205	"{"log.level":"error","@timestamp":"2023-08-14T05:49:34.205Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691992234266	"{"log.level":"error","@timestamp":"2023-08-14T05:50:34.266Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691992264295	"{"log.level":"error","@timestamp":"2023-08-14T05:51:04.295Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691992324355	"{"log.level":"error","@timestamp":"2023-08-14T05:52:04.355Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691992354388	"{"log.level":"error","@timestamp":"2023-08-14T05:52:34.387Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691992384415	"{"log.level":"error","@timestamp":"2023-08-14T05:53:04.415Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"
1691992414426	"{"log.level":"error","@timestamp":"2023-08-14T05:53:34.426Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.18.113:18602/intake/v2/events\": local error: tls: bad record MAC
"}"

Carson_Ip · August 15, 2023, 1:47pm

Hi @lalitprasanth

Can you please provide the apm-server config (especially the TLS related part) and apm-server logs at the same time period? Please remove any sensitive information.
Is the APM server TLS settings configured properly? The agent logs show that there are tls errors. Would be good to see what the apm-server logs show.
Is there any proxy / load balancer between the APM agent and APM server?

Using this configuration we were able to trace all our applications.

Do you mean that some APM features work, while it is showing all these errors? i.e. it is not completely broken

Thanks.

lalitprasanth · August 21, 2023, 11:54am

Hi Carson,

We have installed the apm server directly on elastic cloud integrations and have done no additional config for apm server.

After installing this we followed the apm agent installation with the nodejs app and started sending traces to apm server.

There is additional proxy between agent and apm server

Carson_Ip · August 21, 2023, 12:19pm

Hi @lalitprasanth ,

Please check apm-server logs as well as proxy logs to see if there are any errors.
If possible, try sending directly from agent to apm-server without additional proxy, to eliminate the possibility of a proxy TLS misconfiguration.

Thanks,
Carson

lalitprasanth · August 21, 2023, 12:32pm

Hi Carson,

Not sure but I am seeing no logs for the agent, it says logs were disabled.

Also we don't have any proxy, we have our applications running on AWS ECS, which has all traffic outbound access on the load balancer security group.

Carson_Ip · August 21, 2023, 2:28pm

I forgot to ask this, but how is the apm-server deployed? Is it on Elastic Cloud, or is it self-hosted but under fleet?

If it is Elastic Cloud, can you please send me the deployment id in a private message, so I can check if there are any glaring errors?

If it is self-hosted, you may be able to access the logs from the apm-server container. When posting the logs on discuss forum, please remove any sensitive information.

In the 1st response you said "There is additional proxy between agent and apm server", but also "Also we don't have any proxy" and "load balancer" in the previous message. Can you please clarify whether you have a proxy/LB between applications and apm-server?

Thanks.

lalitprasanth · August 22, 2023, 4:41am

Hi Carson,

We are using Elastic Cloud, I've shared the deployment Id in you DM.

In the 1st response you said "There is additional proxy between agent and apm server", but also "Also we don't have any proxy" and "load balancer" in the previous message. Can you please clarify whether you have a proxy/LB between applications and apm-server?

I am very sorry to confuse you, its typo, "There is no* additional proxy between agent and apm server".

Carson_Ip · August 22, 2023, 10:37am

Thanks for the DM. Could you please try disabling TLS 1.3 in integration settings and check if the problem persists?

lalitprasanth · August 22, 2023, 10:49am

Hi Carson,

Unable to toggle to disable.. Am I checking the correct place?

Carson_Ip · August 22, 2023, 11:01am

Sorry for the confusion, just enter TLSv1.1 and TLSv1.2 in the supported protocol versions:

lalitprasanth · August 22, 2023, 11:12am

Hi Carson,

I have made the TLS changes and saved the agent settings, tried to redeploy one of our application but still I am seeing the errors. Attaching the logs for your reference

1692702421621	{"log.level":"info","@timestamp":"2023-08-22T11:07:01.620Z","log":{"logger":"elastic-apm-node"},"agentVersion":"3.48.0","env":{"pid":1,"proctitle":"node","os":"linux 5.10.184-175.749.amzn2.x86_64","arch":"x64","host":"ip-10-180-13-208.ap-south-1.compute.internal","timezone":"UTC+00","runtime":"Node.js v14.21.3"},"config":{"serviceName":{"source":"start","value":"fasal-fresh","commonName":"service_name"},"serviceVersion":{"source":"default","value":"1.0.0","commonName":"service_version"},"serverUrl":{"source":"start","value":"https://e7dd85f445ed466cbefbba7102e42d39.apm.ap-south-1.aws.elastic-cloud.com/","commonName":"server_url"},"logLevel":{"source":"start","value":"info","commonName":"log_level"},"secretToken":{"source":"start","value":"[REDACTED]","commonName":"secret_token"},"ignoreUrls":{"source":"start","sourceValue":["/","/metrics"]},"captureBody":{"source":"start","value":"transactions","commonName":"capture_body"},"apiRequestSize":{"source":"start","value":1048576,"sourceValue":"1mb"},"environment":{"source":"start","value":"qa"}},"activationMethod":"require","ecs":{"version":"1.6.0"},"message":"Elastic APM Node.js Agent v3.48.0"}
1692702429034	(node:1) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 uncaughtException listeners added to [process]. Use emitter.setMaxListeners() to increase limit
1692702429034	(Use `node --trace-warnings ...` to show where the warning was created)
1692702429203	(node:1) [MONGOOSE] DeprecationWarning: Mongoose: the `strictQuery` option will be switched back to `false` by default in Mongoose 7. Use `mongoose.set('strictQuery', false);` if you want to prepare for this change. Or use `mongoose.set('strictQuery', true);` to suppress this warning.
1692702429204	(node:1) NOTE: The AWS SDK for JavaScript (v2) will be put into maintenance mode in 2023.
1692702429204	Please migrate your code to use AWS SDK for JavaScript (v3).
1692702429204	For more information, check the migration guide at https://a.co/7PzMCcy
1692702429207	{"log.level":"warn","@timestamp":"2023-08-22T11:07:09.206Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"cloud metadata requests timed out, using default values instead"}
1692702429314	{"log.level":"info","@timestamp":"2023-08-22T11:07:09.313Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"Central config success: updated transactionMaxSpans: 10000"}
1692702429314	{"log.level":"info","@timestamp":"2023-08-22T11:07:09.314Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"Central config success: updated spanStackTraceMinDuration: -1"}
1692702429544	{"env":"qa","label":"general-logger","level":"info","message":"🚀 Server ready at http://localhost:4000/graphql","source":"fasal-fresh-server","timestamp":"2023-08-22T11:07:09.541Z","type":"app-log"}
1692702439603	"{"log.level":"error","@timestamp":"2023-08-22T11:07:19.603Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.13.154:18372/intake/v2/events\": use of closed network connection
"}"
1692702461832	"{"log.level":"error","@timestamp":"2023-08-22T11:07:41.832Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.13.154:18372/intake/v2/events\": local error: tls: bad record MAC
"}"
1692702593182	"{"log.level":"error","@timestamp":"2023-08-22T11:09:53.182Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (502): Unexpected APM Server response
Post \"https://172.22.13.154:18372/intake/v2/events\": local error: tls: bad record MAC
"}"

Carson_Ip · August 22, 2023, 12:30pm

Hi @lalitprasanth ,

Could you please raise a support ticket? In the ticket we would be able to dive into deployment specifics.

Thanks.

lalitprasanth · August 23, 2023, 5:17am

Hi Carson,

Unfortunately we don't have technical support plan for our account

Is there any way I can provide any details you need to investigate further, or join a quick call.

Carson_Ip · August 23, 2023, 10:28am

Hi @lalitprasanth ,

I did a bit of research and this does not seem like an APM server-specific problem, but rather a generic network/hardware issue. Therefore, I'm afraid I can't provide any further help other than to attach a link to an example fix: GO TLS issue: remote error: tls: bad record MAC · Issue #1774 · rclone/rclone · GitHub . Hope that works for you.

Thanks,
Carson

system · September 20, 2023, 10:28am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.