Issue with APM integration with Meteor application

Kesha_Shah · July 20, 2023, 8:46am

Kibana version: 8.8.1

Elasticsearch version: 8.8.1

APM Server version: 8.8.1

APM Agent language and version: Using NodeJS agent for Meteor App

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

We are trying to integrate elastic-apm with meteor app but seems like there are multiple issues with it.
I have already checked previous discussion on meteor with plugin meteor-elastic-apm but I dont think its compatible with Meteor 2.8 version.

I have removed plugin from my application and just added nodeJs elastic agent, but when I try to create transaction manually inside Meteor Method, it creates traceId but eventually it crashes. Below are few debug and crash logs

I20230720-13:03:11.007(5.5)? {"log.level":"debug","@timestamp":"2023-07-20T07:33:11.006Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"start trace {\"trans\":\"0e9a29d467ccc26a\",\"trace\":\"e462aae1e19d9d573dff2ed8b295d093\",\"name\":\"unnamed\",\"type\":\"internal\",\"subtype\":null,\"action\":null}"}
I20230720-13:03:11.007(5.5)? {"log.level":"debug","@timestamp":"2023-07-20T07:33:11.007Z","log":{"logger":"elastic-apm-node"},"ctxmgr":"AsyncLocalStorageRunContextManager( RunContext<Transaction(0e9a29, 'forecast-get')> )","ecs":{"version":"1.6.0"},"message":"supersedeWithTransRunContext(<Trans 0e9a29d467ccc26a>)"}
I20230720-13:03:11.068(5.5)? {"log.level":"debug","@timestamp":"2023-07-20T07:33:11.068Z","log":{"logger":"elastic-apm-node"},"ctxmgr":"AsyncLocalStorageRunContextManager( RunContext<> )","ecs":{"version":"1.6.0"},"message":"addEndedTransaction(forecast-get)"}
I20230720-13:03:11.069(5.5)? {"log.level":"debug","@timestamp":"2023-07-20T07:33:11.068Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"sending transaction {\"trans\":\"0e9a29d467ccc26a\",\"trace\":\"e462aae1e19d9d573dff2ed8b295d093\"}"}
I20230720-13:03:11.069(5.5)? {"log.level":"debug","@timestamp":"2023-07-20T07:33:11.068Z","log":{"logger":"elastic-apm-node"},"trans":"0e9a29d467ccc26a","name":"forecast-get","trace":"e462aae1e19d9d573dff2ed8b295d093","type":"internal","result":"success","duration":61.813,"ecs":{"version":"1.6.0"},"message":"ended transaction"}
W20230720-13:03:11.117(5.5)? (STDERR) TypeError: Cannot read property 'Symbol(kResourceStore)' of undefined
W20230720-13:03:11.119(5.5)? (STDERR)     at AsyncLocalStorage._propagate (async_hooks.js:288:34)
W20230720-13:03:11.119(5.5)? (STDERR)     at AsyncHook.init (async_hooks.js:255:22)
W20230720-13:03:11.120(5.5)? (STDERR)     at emitInitNative (internal/async_hooks.js:203:43)
W20230720-13:03:11.120(5.5)? (STDERR)     at emitInitScript (internal/async_hooks.js:493:3)
W20230720-13:03:11.120(5.5)? (STDERR)     at initAsyncResource (internal/timers.js:160:5)
W20230720-13:03:11.120(5.5)? (STDERR)     at new Timeout (internal/timers.js:194:3)
W20230720-13:03:11.121(5.5)? (STDERR)     at setUnrefTimeout (internal/timers.js:376:17)
W20230720-13:03:11.121(5.5)? (STDERR)     at Socket.setStreamTimeout [as setTimeout] (internal/stream_base_commons.js:263:22)
W20230720-13:03:11.121(5.5)? (STDERR)     at IncomingMessage.setTimeout (_http_incoming.js:99:15)
W20230720-13:03:11.121(5.5)? (STDERR)     at Server.WebApp._timeoutAdjustmentRequestCallback (packages/webapp/webapp_server.js:273:7)
W20230720-13:03:11.121(5.5)? (STDERR)     at packages/ddp-server/stream_server.js:184:23
W20230720-13:03:11.121(5.5)? (STDERR)     at Array.forEach (<anonymous>)
W20230720-13:03:11.122(5.5)? (STDERR)     at Function._.each._.forEach (packages/underscore.js:139:11)
W20230720-13:03:11.122(5.5)? (STDERR)     at Server.newListener (packages/ddp-server/stream_server.js:183:11)
W20230720-13:03:11.124(5.5)? (STDERR)     at Server.emit (events.js:400:28)
W20230720-13:03:11.125(5.5)? (STDERR)     at Server.emit (domain.js:475:12)
W20230720-13:03:11.125(5.5)? (STDERR)     at parserOnIncoming (_http_server.js:900:12)
W20230720-13:03:11.125(5.5)? (STDERR)     at HTTPParser.parserOnHeadersComplete (_http_common.js:127:17)
W20230720-13:03:11.125(5.5)? (STDERR) TypeError: Cannot read property 'Symbol(kResourceStore)' of undefined
W20230720-13:03:11.126(5.5)? (STDERR)     at AsyncLocalStorage._propagate (async_hooks.js:288:34)
W20230720-13:03:11.126(5.5)? (STDERR)     at AsyncHook.init (async_hooks.js:255:22)
W20230720-13:03:11.126(5.5)? (STDERR)     at emitInitNative (internal/async_hooks.js:203:43)
W20230720-13:03:11.127(5.5)? (STDERR)     at emitInitScript (internal/async_hooks.js:493:3)
W20230720-13:03:11.127(5.5)? (STDERR)     at promiseInitHook (internal/async_hooks.js:323:3)
W20230720-13:03:11.127(5.5)? (STDERR)     at Function.reject (<anonymous>)
W20230720-13:03:11.127(5.5)? (STDERR)     at process.<anonymous> (packages/inter-process-messaging.js:227:47)
W20230720-13:03:11.127(5.5)? (STDERR)     at process.emit (events.js:412:35)
W20230720-13:03:11.127(5.5)? (STDERR)     at process.emit (domain.js:475:12)
W20230720-13:03:11.127(5.5)? (STDERR)     at process.exit (internal/process/per_thread.js:179:15)
W20230720-13:03:11.127(5.5)? (STDERR)     at fatalError (internal/async_hooks.js:176:11)
W20230720-13:03:11.127(5.5)? (STDERR)     at emitInitNative (internal/async_hooks.js:210:5)
W20230720-13:03:11.127(5.5)? (STDERR)     at emitInitScript (internal/async_hooks.js:493:3)
W20230720-13:03:11.127(5.5)? (STDERR)     at initAsyncResource (internal/timers.js:160:5)
W20230720-13:03:11.127(5.5)? (STDERR)     at new Timeout (internal/timers.js:194:3)
W20230720-13:03:11.127(5.5)? (STDERR)     at setUnrefTimeout (internal/timers.js:376:17)
W20230720-13:03:11.127(5.5)? (STDERR)     at Socket.setStreamTimeout [as setTimeout] (internal/stream_base_commons.js:263:22)
W20230720-13:03:11.127(5.5)? (STDERR)     at IncomingMessage.setTimeout (_http_incoming.js:99:15)
W20230720-13:03:11.127(5.5)? (STDERR)     at Server.WebApp._timeoutAdjustmentRequestCallback (packages/webapp/webapp_server.js:273:7)
W20230720-13:03:11.127(5.5)? (STDERR)     at packages/ddp-server/stream_server.js:184:23
W20230720-13:03:11.127(5.5)? (STDERR)     at Array.forEach (<anonymous>)
W20230720-13:03:11.127(5.5)? (STDERR)     at Function._.each._.forEach (packages/underscore.js:139:11)
W20230720-13:03:11.128(5.5)? (STDERR)     at Server.newListener (packages/ddp-server/stream_server.js:183:11)
W20230720-13:03:11.128(5.5)? (STDERR)     at Server.emit (events.js:400:28)
W20230720-13:03:11.128(5.5)? (STDERR)     at Server.emit (domain.js:475:12)
W20230720-13:03:11.128(5.5)? (STDERR)     at parserOnIncoming (_http_server.js:900:12)
W20230720-13:03:11.128(5.5)? (STDERR)     at HTTPParser.parserOnHeadersComplete (_http_common.js:127:17)
=> Exited with code: 1
=> Your application is crashing. Waiting for file change.

This is my code to start transaction

Meteor.methods({
  getAllFarm() {
      this.unblock();
      const trans = apm.startTransaction(`Get-farm`, 'internal');
      console.log(`get all farms`);
      console.log(`current transaction is ${apm.currentTransaction}`);
      const user = Meteor.user();
      // Some code 
      if (trans) trans.end();
    }
})

Please note by default there is no transaction created as its not HTTP call and it works via pub/sub mechanism.
Also with meteor-elastic-apm plugin it was giving me same error.

Any help is appreciated to fix this issue

David_Luna_Bistuer · July 20, 2023, 4:31pm

Hi @Kesha_Shah

That missing symbol is not from the nodejs agent so probably the instrumentation is involved. Thanks for the log and the code snippet but I'm not familiar with the framework and I need some extra info:

which method did you use for instrumentation. Take a look at this page for reference
could you provide a small repro of the issue? You can create a PR to this todo App to add the changes that lead to this error.

Cheers!

trentm · August 10, 2023, 6:58pm

@Kesha_Shah If you are still hitting this, what version of Node.js are you using?

Your stack trace looks like this core Node.js issue:

github.com/nodejs/node

AsyncLocalStorage: Cannot read property 'Symbol(kResourceStore)'

opened 07:06PM - 29 Jul 20 UTC

closed 07:28AM - 03 Aug 20 UTC

johanneswuerbach

confirmed-bug async_hooks

* **Version**: v14.6.0 (works on v14.5.0) * **Platform**: Unix * **Subsystem**: async_hooks ### What steps will reproduce the bug?  I'm not entirely sure yet, but will continue investigating. Any pointers would be appreciated. Sadly the code itself is private, but I'll try to extract a reproduction. ### How often does it reproduce? Is there a required condition? I only see this crash reliably on every test run on one of our services, all the others seem to work fine. The service works on v14.4.0 and v14.5.0, so this seems to be a regression introduced in v14.6.0. ### What is the expected behavior? Working AsyncLocalStorage. ### What do you see instead? An exception ``` TypeError: Cannot read property 'Symbol(kResourceStore)' of undefined at AsyncLocalStorage._propagate (async_hooks.js:246:34) at AsyncHook.init (async_hooks.js:222:22) at emitInitNative (internal/async_hooks.js:198:43) at emitInitScript (internal/async_hooks.js:466:3) at initAsyncResource (internal/timers.js:155:5) at new Timeout (internal/timers.js:188:3) at setUnrefTimeout (internal/timers.js:370:17) at cache (internal/http.js:27:3) at utcDate (internal/http.js:19:18) at ServerResponse._storeHeader (_http_outgoing.js:391:26) at ServerResponse.writeHead (_http_server.js:313:8) at ServerResponse._implicitHeader (_http_server.js:240:8) at write_ (_http_outgoing.js:663:9) at ServerResponse.end (_http_outgoing.js:776:5) at /app/node_modules/@opentelemetry/plugin-http/build/src/http.js:213:87 at HttpPlugin._safeExecute (/app/node_modules/@opentelemetry/plugin-http/build/src/http.js:303:20) at ServerResponse.response.end (/app/node_modules/@opentelemetry/plugin-http/build/src/http.js:213:49) ``` Likely the result of `executionAsyncResource()` returning `undefined` here: https://github.com/nodejs/node/blob/13c5a1629cd025ba560f34f6d3190b2f38d184d4/lib/async_hooks.js#L219 ### Additional information AsyncLocalStorage is used using https://github.com/open-telemetry/opentelemetry-js/tree/v0.10.1 and via the `AsyncLocalStorageContextManager` https://github.com/open-telemetry/opentelemetry-js/blob/v0.10.1/packages/opentelemetry-context-async-hooks/src/AsyncLocalStorageContextManager.ts

I believe that was resolved in Node.js 14.8.0.

system · September 7, 2023, 6:59pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.