Azure Functions nodejs not showing transaction samples

[shevkaboss]

Good day, I'm new to ELK stack and trying to setup APM transactions on my BE application.

We are using serverless infrastructure with Azure Functions and nodejs.

I cannot see any transaction samples in APM view:

Steps to reproduce:

1. Download apm-agent-nodejs/examples/an-azure-function-app at main · elastic/apm-agent-nodejs · GitHub
2. Setup required configuration (apm server url and token)
3. Trigger any endpoint

Am I expecting something from from this?

P.S one of our functions runs on graphql and transactions logged properly. This reproduces for any other function, regardless of what's inside of it.

P.S.2 enabling tracing for apm shows that no spans being captured. Starting and ending span manually does not solve the issue

P.S.3 This reproduces with any version on npm packages and runtime.

[Francesco_Gualazzi]

Hello @shevkaboss thanks for reporting this!

May I ask you to clarify what setup of APM Server are you using?
For example, are you hosting the Elastic stack yourself or are you running it in cloud.elastic.co?

Can you also specify the versions for both the client and server part that you are using?
Cheers

[shevkaboss]

We are running it on Cloud instance via Azure (yes, cloud.elastic.co).

Server instance is 8.12.2:

By the client part you mean version on npm in our case? The last attempt was on the demo provided by ELK: "elastic-apm-node": "^3.42.0".

It isn't working on latest as well: "elastic-apm-node": "^4.7.1"

Thank you for quick reply!

[Francesco_Gualazzi]

Hello

I was trying to figure what's wrong and re-created a similar setup but I could not face any issues.

Can you please clarify what you mean with "one of our functions runs on graphql and transactions logged properly."?
Does this mean that you see traces data populated in the APM UI for this function?

Also when you say "Starting and ending span manually does not solve the issue", do you mean adding instrumentation in the code, in form of spans?

I guess you already validated that the environment variables needed to configure the agent are populated equally in all functions?

Cheers

[shevkaboss]

So the issue were found trying to inject APM to enterprise application.

It runs back-end on nodejs azure functions. There are around 10 of them.

The main data provider function under the hood is graphql server:

Traces appears for all graphql transactions.

On the other hand, we have couple of http trigger and time trigger. Traces from them aren't appearing on APM UI.

Regarding spans. Logging traces from apm I noticed that spans for Graphql function transaction were created during it's execution (even if there are no external dependencies on the code).
On the other hand, there was no spans for http/time functions transaction. I found that this can cause a problem as transaction will not display traces if there are not spans under it.

I've tried to add spans manually by calling apm.startSpan and endSpan during code execution. Even this span appears on logs (logs from local terminal), traces for transactions are still not display on APM UI.

I've also tried locally using CommonJS modules instead of ECMAscript (from docs I noticed that can be a problem), that didn't work.

ENV variables are the same for all functions as the code start on entry point from package.json

Do you have a working copy? I want to try find the differences.
My node version is 18.20.2, tried different versions on Azure Function runtime. On main env we are using "version": "[3.*, 4.0.0)".

P.S. From the screenshot above you can see a workaround creating require for ECMAscript module. This is in order to go through the bug in apm not capturing graphql requests properly (they are displayed as regular http request but not graphql methods. require-in-the-middle needs to be executed properly, but import-in-the-middle looks to be broken).

[trentm (Trent Mick)]

@shevkaboss

I've also tried locally using CommonJS modules instead of ECMAscript (from docs I noticed that can be a problem), that didn't work.

Hi @shevkaboss,

I work on the Node.js APM agent. Are you able to show a small example with code that reproduces the issue. If you run with ELASTIC_APM_LOG_LEVEL=trace that will provide helpful details on what the APM agent is doing.

I'm a little confused on whether we are talking about tracing HTTP or Time triggered Azure Functions, or something you are able to run locally to reproduce the problem.

Thanks.

[shevkaboss]

Traces does not appear on your node-js examples. Tracing not working for neither HTTP nor Time trigger Azure Function.
My steps to reproduce:

1. Clone apm-agent-nodejs/examples/an-azure-function-app at main · elastic/apm-agent-nodejs · GitHub
2. Change initapm.js to contain following:
   require('elastic-apm-node').start({ secretToken: ''/*secret token*/, serverUrl: ''/*service url*/, serviceName: 'test-topic', environment: 'local-test-app', logLevel: 'trace' })
3. run yarn install and yarn start
4. trigger http function by calling GET http://localhost:7071/api/Bye

Results are:

1. Transactions appearing in Trace view. Then, no more details/traces for it (sorry for merged screenshot, Im not allowed to upload more then one image):
   

   

2. Traces for APM doesn't contain a lot of, the only things are:

• start trace
• setting default transcation name
• setting transaction result
• sending transaction

4. Share me your email if you want to have a full logs (or let me know any other way to share them with you, I don't want to post them there lmao)
5. Make a note, no spans created during this transaction. Manually creating spans (by apm.startSpan and endSpan) doesn't work. I except it working without any manual interactions.

P.S demo repo seems to be a little outdated. Latest npm version doesn't work either.

[shevkaboss]

Good day!
Any updates on that?

[shevkaboss]

It's been a while since this issue was opened and my team still experiences the problem.

Transaction samples are not available on APM UI but same UI shows that N total transactions available.

The only transaction samples available for one of my Azure Functions which uses GraphQL, others (http trigger, time trigger and others) do not work.

Can anyone support with that?

[trentm (Trent Mick)]

@shevkaboss Sorry for not having replying a long time.

In one of your messages in Aug you said:

@shevkaboss

Traces does not appear on your node-js examples.

but then in the same message said:

@shevkaboss

Transactions appearing in Trace view.

So I am a little confused. That does show that tracing data for this example Azure Function was captured, right?

I'm not sure what is happening with the "The selected trace cannot be found." shown in your screenshot.

@shevkaboss

Make a note, no spans created during this transaction.

When calling GET http://localhost:7071/api/Bye I don't expect to see any spans: just the one top-level transaction. The Hi endpoint should show spans, however.

Here is my reproduction:

cd apm-agent-nodejs/examples/an-azure-function-app
nvm use 20  # I ran into an IPv6 issue using Node.js v18.
npm install
export ELASTIC_APM_SERVER_URL=...
export ELASTIC_APM_SECRET_TOKEN=...
npm start

curl http://localhost:7071/api/Hi

The func start (from npm start above) logged this:

[2025-01-16T17:30:47.290Z] Executing 'Functions.Hi' (Reason='This function was programmatically called via the host APIs.', Id=264dfc96-7a2c-4876-b7b6-aa7be19ef746)
[2025-01-16T17:30:47.311Z] Executing 'Functions.Bye' (Reason='This function was programmatically called via the host APIs.', Id=d022f079-2441-451b-ba28-b3271970dcea)
[2025-01-16T17:30:47.335Z] Executed 'Functions.Bye' (Succeeded, Id=d022f079-2441-451b-ba28-b3271970dcea, Duration=23ms)
[2025-01-16T17:30:47.342Z] Executed 'Functions.Hi' (Succeeded, Id=264dfc96-7a2c-4876-b7b6-aa7be19ef746, Duration=63ms)

And in Kibana I see this trace:

@shevkaboss

4. Share me your email if you want to have a full logs

You can send logs to trent.mick@elastic.co.

[trentm (Trent Mick)]

@shevkaboss

P.S demo repo seems to be a little outdated.

I agree with that. I am currently working on adding support for Azure Functions version 4 of the Node.js programming model. Migrate to v4 of the Node.js model for Azure Functions | Microsoft Learn

[shevkaboss]

@trentm thanks for reply. Yes, from your screenshot there was a big difference on what was appearing on my ELK instance.

Finally, after another round of diving deeper in this problem I found out misconfigured apm-traces@custom ingest pipeline for APM index. It was producing errors trying to read non-graphql logs which resulted in not showing transaction samples on APM UI.

Simply disabling custom ingest pipeline made everything work like a charm.

Thank you for support!