# Elastic-apm-node memory leak **URL:** \ **Category:** APM\ **Tags:** nodejs\ **Created:** [May 1, 2019, 7:52pm UTC](https://discuss.elastic.co/t/elastic-apm-node-memory-leak/179245 "2019-05-01T19:52:38Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![sammaxwell](https://avatars.discourse-cdn.com/v4/letter/s/a3d4f5/32.png) [@sammaxwell](https://discuss.elastic.co/u/sammaxwell)\ **Post date:** [May 1, 2019, 7:52pm UTC](https://discuss.elastic.co/t/elastic-apm-node-memory-leak/179245/1 "2019-05-01T19:52:39Z")
Ever since I incorporated APM (elastic-apm-node) in some nodejs microservices they have developed memory leaks. I have a small repo that shows a couple strange things that might be related... > **[SamMaxwell/elastic-apm-test](https://github.com/SamMaxwell/elastic-apm-test)** > > tests the Elastic APM service. Contribute to SamMaxwell/elastic-apm-test development by creating an account on GitHub. You will need a MS Sql Server db connection string and an APM server token In the trace you will notice that even though some spans are sent, not all are. Many times I only get one. If you set count to 10 or higher you will also see one or more "MaxListenersExceededWarning: Possible EventEmitter memory leak detected." messages. If you set the count to 4000 you will (hopefully) get: "MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 infoMessage listeners added." "MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 errorMessage listeners added." and then ... **"FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory"** **Kibana version** : 6.7.1 **Elasticsearch version** : 6.7.1 **APM Server version** : 6.6.1 **APM Agent language and version** : elastic-apm-node 2.10.0 **Steps to reproduce** : ``` First Case: not all spans sent and memory leak warning ``` 1. clone [https://github.com/SamMaxwell/elastic-apm-test.git](https://github.com/SamMaxwell/elastic-apm-test.git) 2. cd into the new folder 3. npm install 4. set ELASTIC\_APM\_SERVER\_URL, ELASTIC\_APM\_SECRET\_TOKEN, ELASTIC\_APM\_LOG\_LEVEL env variables 5. node . "some MS Sql Server db connection string" "some query" 10 6. inspect the trace output 7. delete the ELASTIC\_APM\_LOG\_LEVEL env var 8. node . "some MS Sql Server db connection string" "some query" 4000 9. look at the FATAL ERROR 10. set NO\_APM env var to anything (to disable APM) 11. node . "some MS Sql Server db connection string" "some query" 4000 12. notice how it works with no FATAL ERROR ---
**Author:** ![stephenbelanger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenbelanger/32/34880_2.png) [@stephenbelanger](https://discuss.elastic.co/u/stephenbelanger)\ **Post date:** [May 3, 2019, 5:55pm UTC](https://discuss.elastic.co/t/elastic-apm-node-memory-leak/179245/2 "2019-05-03T17:55:31Z")
Thanks for the report and sorry you're having issues. I've cloned your repo and tried running it locally--I _do_ get the emitter count warning, but no fatal error, even bumping it up all the way to 1,000,000. It's likely just caused by the closure leak caused by the event handlers though. It's due to a mismatch between adding an event handler and trying to remove it due to the function wrap in [https://github.com/elastic/apm-agent-nodejs/blob/master/lib/instrumentation/index.js#L284-L299](https://github.com/elastic/apm-agent-nodejs/blob/master/lib/instrumentation/index.js#L284-L299) ---
**Author:** ![sammaxwell](https://avatars.discourse-cdn.com/v4/letter/s/a3d4f5/32.png) [@sammaxwell](https://discuss.elastic.co/u/sammaxwell)\ **Post date:** [May 3, 2019, 6:20pm UTC](https://discuss.elastic.co/t/elastic-apm-node-memory-leak/179245/3 "2019-05-03T18:20:00Z")
Thanks for looking into this. Maybe getting the FATAL error somehow involves the nature/structure of data returned from the query. I was returning between 1 and 100 rows from a table with more than 50 columns in my testing. Are you suggesting I do something to the application logic? ---
**Author:** ![stephenbelanger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenbelanger/32/34880_2.png) [@stephenbelanger](https://discuss.elastic.co/u/stephenbelanger)\ **Post date:** [May 3, 2019, 6:22pm UTC](https://discuss.elastic.co/t/elastic-apm-node-memory-leak/179245/4 "2019-05-03T18:22:38Z")
It's possible, but no need to do anything right now. I've got an idea for a fix for the emitter leak in mind. I'll try to get that out hopefully later today. After that, give it a try and see if that fixes the fatal error too--I suspect it does, but we'll see. ---
**Author:** ![stephenbelanger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenbelanger/32/34880_2.png) [@stephenbelanger](https://discuss.elastic.co/u/stephenbelanger)\ **Post date:** [May 3, 2019, 7:56pm UTC](https://discuss.elastic.co/t/elastic-apm-node-memory-leak/179245/5 "2019-05-03T19:56:38Z")
I've got a pull request here, just running through the CI now. I'll let you know when I land and release that. [https://github.com/elastic/apm-agent-nodejs/pull/1044](https://github.com/elastic/apm-agent-nodejs/pull/1044) ---
**Author:** ![sammaxwell](https://avatars.discourse-cdn.com/v4/letter/s/a3d4f5/32.png) [@sammaxwell](https://discuss.elastic.co/u/sammaxwell)\ **Post date:** [May 3, 2019, 8:07pm UTC](https://discuss.elastic.co/t/elastic-apm-node-memory-leak/179245/6 "2019-05-03T20:07:37Z")
thanks! ---
**Author:** ![stephenbelanger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenbelanger/32/34880_2.png) [@stephenbelanger](https://discuss.elastic.co/u/stephenbelanger)\ **Post date:** [May 3, 2019, 10:08pm UTC](https://discuss.elastic.co/t/elastic-apm-node-memory-leak/179245/7 "2019-05-03T22:08:18Z")
I've got the fix out in 2.11.0. Give that a try and let me know how it goes. 🙂 ---
**Author:** ![sammaxwell](https://avatars.discourse-cdn.com/v4/letter/s/a3d4f5/32.png) [@sammaxwell](https://discuss.elastic.co/u/sammaxwell)\ **Post date:** [May 6, 2019, 12:32pm UTC](https://discuss.elastic.co/t/elastic-apm-node-memory-leak/179245/8 "2019-05-06T12:32:54Z")
It seems like 2.11.0 fixes the MaxListeners warning and the FATAL out of memory! I still see that the SPANS are not being automatically created except for the first iteration. - set ELASTIC\_APM\_LOG\_LEVEL to "Trace" - run the test for 10 iterations - look at the trace output - notice how the "start span"/"ended span"/"encoding span" appears on the first iteration, but not the 2nd through 10th iterations - if you look at the apm docs with kibana you will notice that the spans do not appear ---
**Author:** ![stephenbelanger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenbelanger/32/34880_2.png) [@stephenbelanger](https://discuss.elastic.co/u/stephenbelanger)\ **Post date:** [May 7, 2019, 4:55pm UTC](https://discuss.elastic.co/t/elastic-apm-node-memory-leak/179245/9 "2019-05-07T16:55:05Z")
It's a context loss issue. We use async\_hooks to maintain context through an async execution tree, but it appears to not support thenables, which is an object that looks like a promise by having a "then" method, but is not _actually_ a promise. Tedious seems to use thenables internally. After awaiting a thenable async\_hooks loses track of what the async ids are. A way around it is to wrap it in a _real_ promise before awaiting, like this: ``` await new Promise((...args) => { promiseLike.then(...args) }) ``` ---
**Author:** ![sammaxwell](https://avatars.discourse-cdn.com/v4/letter/s/a3d4f5/32.png) [@sammaxwell](https://discuss.elastic.co/u/sammaxwell)\ **Post date:** [May 7, 2019, 6:29pm UTC](https://discuss.elastic.co/t/elastic-apm-node-memory-leak/179245/10 "2019-05-07T18:29:14Z")
I opened a second topic [Not all transactions/spans are sent from agent / stored in elastic and worse when using addLabels](https://discuss.elastic.co/t/not-all-transactions-spans-are-sent-from-agent-stored-in-elastic-and-worse-when-using-addlabels/179961/7) when I thought it might make more sense to do so. ---
**Author:** ![sammaxwell](https://avatars.discourse-cdn.com/v4/letter/s/a3d4f5/32.png) [@sammaxwell](https://discuss.elastic.co/u/sammaxwell)\ **Post date:** [May 7, 2019, 6:49pm UTC](https://discuss.elastic.co/t/elastic-apm-node-memory-leak/179245/11 "2019-05-07T18:49:51Z")
I use knex in the application, not tedious. tedious is used by mssql, which is the client specified in the config sent to knex to get the query-builder. what are you suggesting I wrap with a Promise? ---
**Author:** ![stephenbelanger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenbelanger/32/34880_2.png) [@stephenbelanger](https://discuss.elastic.co/u/stephenbelanger)\ **Post date:** [May 7, 2019, 11:40pm UTC](https://discuss.elastic.co/t/elastic-apm-node-memory-leak/179245/12 "2019-05-07T23:40:08Z")
Oh, sorry--I meant knex, not tedious. I'm suggesting you wrap your queries that you want to `await` in a real promise so `async_hooks` can propagate the context properly. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 28, 2019, 7:41pm UTC](https://discuss.elastic.co/t/elastic-apm-node-memory-leak/179245/13 "2019-05-28T19:41:44Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.