APM transactions from one specific service aren't shipping

[michaelbdavidson7 (Michael Davidson)]

Elastic cloud running 7.0

APM Server version:
Whichever comes with EC 7.0, which

APM Agent language and version:
npm version - elastic-apm-node v2.13.0

Issue:
Hi, we're using elastic cloud APM in our four applications. Three are sending logs fine, but the fourth is not sending transactions specifically. It is sending errors and non-transaction-related metrics just fine. I deleted the index in kibana a while ago and it was around then that it stopped getting those transactions. Could it just have never recovered ? is there a way to fix this from kibana?

Additional info: I went to the Discover page in kibana and could see everything except for the transactions from that fourth service.

Thanks guys.

[axw (Andrew Wilkins)]

Hi @michaelbdavidson7, welcome to the forum!

Have you already enabled trace logging? If not, I'd recommend following https://www.elastic.co/guide/en/apm/agent/nodejs/current/troubleshooting.html#debug-mode to do that, and look out for any error responses from the server. That might give us some more clues.

[michaelbdavidson7 (Michael Davidson)]

Hi Andrew,
Thanks for the reply. I did this locally, and didn't see any errors thrown. I now see that I don't think some of the APM errors are crossing over either..
What else could it be?
This used to work, I think it was from when I was messing with ES. is there a way I can just reset my ES index or something?

[axw (Andrew Wilkins)]

@michaelbdavidson7

Thanks for the reply. I did this locally, and didn't see any errors thrown. I now see that I don't think some of the APM errors are crossing over either..
What else could it be?
This used to work, I think it was from when I was messing with ES. is there a way I can just reset my ES index or something?

You could blow away the indices again, but I'm not convinced that'll help. I suggest checking the APM Server logs to see if there is any strange showing up there. It seems pretty odd that you would be receiving some but not all errors from the service - unless it's a short-lived service, and it's failing to flush data to the server. But then I wouldn't expect you to be seeing metrics either.

One thing that can cause data to be dropped is when labels have different types across services. If one service is recording transactions with a label foo: <number> and another service records transactions with a label foo: <string>, then the server will fail to record them for one or the other. You should see an error in the agent, though.

[michaelbdavidson7 (Michael Davidson)]

hey this was a code issue on our part, we imported our routes before the APM so the APM wasn't watching them ..
Thanks!

[axw (Andrew Wilkins)]

Ahhh! Glad to hear it's sorted.