APM dashboard shows no data

Elastic Observability APM
,
1

APM dashboard shows no data

image
image3074×1058 243 KB

I can see in the APM index, data is getting loaded.

Kibana logs

{"type":"log","@timestamp":"2022-12-20T19:23:25+00:00","tags":["error","plugins","apm"],"pid":2513,"message":"Error: Not Found\n    at handler (/usr/share/kibana/x-pack/plugins/apm/server/routes/settings/agent_configuration.js:299:27)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)\n    at /usr/share/kibana/x-pack/plugins/apm/server/routes/register_routes/index.js:139:22\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:163:30)\n    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)\n    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)\n    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:279:9) {\n  data: null,\n  isBoom: true,\n  isServer: false,\n  output: {\n    statusCode: 404,\n    payload: { statusCode: 404, error: 'Not Found', message: 'Not Found' },\n    headers: {}\n  }\n}"}

apm-server logs

{"log.level":"error","@timestamp":"2022-12-20T20:09:00.726Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":60},"message":"data validation error","service.name":"apm-server","event.dataset":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-java/1.24.0","source.address":"xx.xx.xx.xx","http.request.id":"a2229383-5df0-44c3-80da-9753d10e9a5f","event.duration":4168122814,"http.response.status_code":400,"error.message":"decode error: data read error: v2.errorRoot.Error: v2.errorEvent.Exception: v2.errorException.Cause: []v2.errorException: v2.errorException.Message: ReadString: expects \" or n,","ecs.version":"1.6.0"}
2

Seeing this error in apm-agent logs

Dec 21 17:51:28 xxxxx java[26334]: 2022-12-21 17:51:28,396 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 0 seconds (+/-10%)
Dec 21 17:51:28 xxxxx java[26334]: 2022-12-21 17:51:28,396 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Unexpected end of file from server, response code is -1
Dec 21 17:51:28 xxxxx java[26334]: 2022-12-21 17:51:28,396 [elastic-apm-server-reporter] WARN  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - null
3

It doesn't look like the agents are sending data to the server, please check from the agent host that you can curl to the server specified by the server_urls config you use

4

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.