# Python client APM very slow when local list variable is very large **URL:** \ **Category:** APM\ **Tags:** python\ **Created:** [July 18, 2022, 5:55pm UTC](https://discuss.elastic.co/t/python-client-apm-very-slow-when-local-list-variable-is-very-large/309935 "2022-07-18T17:55:35Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Kohlman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kohlman/32/87348_2.png) [@Kohlman](https://discuss.elastic.co/u/Kohlman)\ **Post date:** [July 18, 2022, 5:55pm UTC](https://discuss.elastic.co/t/python-client-apm-very-slow-when-local-list-variable-is-very-large/309935/1 "2022-07-18T17:55:35Z")
I've deleted only the relevant portions of your template. **APM Agent language and version** : Python 6.7.2 **Steps to reproduce** : 1. Create a very large list. In our case, we had a list of objects. One of the property's was a list of list. So we essentially had a list of list of list. 2. We noticed then when an our code throws an exception, the elastic python agent then evaluates the local variables. But because this data in one of our particular cases is very large, as in millions of items, the APM python client code was taking up huge amounts of CPU. In addition, it would take over a minute to finally create the APM output to send to the APM server. To simply replicate the issue, I: (see code snipit below) - created a simple script to create a list of list of lists of integers. - in one method I had my "with elasticAPMTransaction:" statement. - under this I created the list (200 items) of list (300 items) of list (40 items) aka 24M items total. - started to iterate through the outer list and picked an arbitrary spot to through an exception. This essentially mimicked the method where the real code threw an exception. - wrapped this whole thing in a profiler and evaluated the output. This would take 60 seconds +/- to complete. Seems like the issue is in the varmap function in elasticapm\utils\__init_\_.py but there maybe lots of ways to solve this issue. The snipit of code looks like: ```auto with elastic_apm_transaction: x = 0 my_list = self.build_lists() for foo in my_list: x += 1 if x >= 20: time_at_exception = datetime.now() print ("ABOUT TO THROW EXCEPTION AT ", time_at_exception.strftime("%A, %H:%M.%S")) raise Exception("KA_BOOM") ``` ---
**Author:** ![basepi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/basepi/32/53056_2.png) [@basepi](https://discuss.elastic.co/u/basepi)\ **Post date:** [July 18, 2022, 8:31pm UTC](https://discuss.elastic.co/t/python-client-apm-very-slow-when-local-list-variable-is-very-large/309935/2 "2022-07-18T20:31:00Z")
We have a [number of tuning options around local variable collection](https://www.elastic.co/guide/en/apm/agent/python/current/configuration.html#config-collect-local-variables). You can turn it off entirely, or just limit the list length for your use case. Note that this only looks at the top level of the list, if I remember correctly, so keep that in mind as you're tuning. Does that solve your issue? ---
**Author:** ![Kohlman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kohlman/32/87348_2.png) [@Kohlman](https://discuss.elastic.co/u/Kohlman)\ **Post date:** [July 19, 2022, 9:11pm UTC](https://discuss.elastic.co/t/python-client-apm-very-slow-when-local-list-variable-is-very-large/309935/3 "2022-07-19T21:11:10Z")
Colton, Thanks for the quick response. I tried setting ELASTIC\_APM\_LOCAL\_VAR\_MAX\_LENGTH to 20 and it seemingly had no affect. I don't want to set ELASTIC\_APM\_COLLECT\_LOCAL\_VARIABLES to False, because we find the variables very useful when there are errors. Any other thoughts? Thanks, Kohlman ---
**Author:** ![basepi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/basepi/32/53056_2.png) [@basepi](https://discuss.elastic.co/u/basepi)\ **Post date:** [July 20, 2022, 2:35pm UTC](https://discuss.elastic.co/t/python-client-apm-very-slow-when-local-list-variable-is-very-large/309935/4 "2022-07-20T14:35:29Z")
[Did you try `ELASTIC_APM_LOCAL_VAR_LIST_MAX_LENGTH`?](https://www.elastic.co/guide/en/apm/agent/python/current/configuration.html#config-local-list-var-max-length) I'm not surprised that `ELASTIC_APM_LOCAL_VAR_MAX_LENGTH` didn't solve your issue as we can't check the length until we've converted the variable to a string (which would be the performance bottleneck for your list of lists of lists). ---
**Author:** ![Kohlman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kohlman/32/87348_2.png) [@Kohlman](https://discuss.elastic.co/u/Kohlman)\ **Post date:** [July 20, 2022, 3:34pm UTC](https://discuss.elastic.co/t/python-client-apm-very-slow-when-local-list-variable-is-very-large/309935/5 "2022-07-20T15:34:25Z")
Very strange. I just tried ELASTIC\_APM\_LOCAL\_VAR\_LIST\_MAX\_LENGTH as you suggested. I set the value to 20 and it went faster. What is strange is after wards I went back to the documentation and is says the default is 10. Setting it to 10 gets it down to 5 seconds. I tried changing the varmap function to only use 20 elements from each list and this completes in less then a second. ---
**Author:** ![basepi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/basepi/32/53056_2.png) [@basepi](https://discuss.elastic.co/u/basepi)\ **Post date:** [July 20, 2022, 3:55pm UTC](https://discuss.elastic.co/t/python-client-apm-very-slow-when-local-list-variable-is-very-large/309935/6 "2022-07-20T15:55:02Z")
You're right, it certainly seems to be 10 items by default 🤔 Does removing the config for `ELASTIC_APM_LOCAL_VAR_LIST_MAX_LENGTH` take it back to long runtimes? > I tried changing the varmap function to only use 20 elements from each list and this completes in less then a second. Very odd. Did you also remove the recursive piece of `varmap()` with this edit? That's the only reason I could see for such a speed-up, since the default is 10..... ---
**Author:** ![Kohlman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kohlman/32/87348_2.png) [@Kohlman](https://discuss.elastic.co/u/Kohlman)\ **Post date:** [July 20, 2022, 4:13pm UTC](https://discuss.elastic.co/t/python-client-apm-very-slow-when-local-list-variable-is-very-large/309935/7 "2022-07-20T16:13:30Z")
Verified: - I reverted varmap change - took out the profiling code. - took out setting ELASTIC\_APM\_LOCAL\_VAR\_LIST\_MAX\_LENGTH took 52 seconds. Tried only only put the ELASTIC\_APM\_LOCAL\_VAR\_LIST\_MAX\_LENGTH = 10 back took 40 seconds. take out ELASTIC\_APM\_LOCAL\_VAR\_LIST\_MAX\_LENGTH, put back varmap change took less then a second Here is another interesting thing, looking at APM for the last one, I still see the lists each with . . . drumroll please . . . 10 items and when it cuts off a list you see the message "X more elements." So you are correct the default is 10, but the perf bug is there, it looks like it evaluates the entire list so it can calculate how many items were truncated. ---
**Author:** ![Kohlman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kohlman/32/87348_2.png) [@Kohlman](https://discuss.elastic.co/u/Kohlman)\ **Post date:** [July 20, 2022, 5:04pm UTC](https://discuss.elastic.co/t/python-client-apm-very-slow-when-local-list-variable-is-very-large/309935/8 "2022-07-20T17:04:08Z")
FYI - The change I made to the varmap function ```auto def varmap(func, var, context=None, name=None, **kwargs): """ Executes ``func(key_name, value)`` on all values, recursively discovering dict and list scoped values. """ if context is None: context = set() objid = id(var) if objid in context: return func(name, "<...>", **kwargs) context.add(objid) if isinstance(var, dict): # iterate over a copy of the dictionary to avoid "dictionary changed size during iteration" issues ret = func( name, dict((k, varmap(func, v, context, k, **kwargs)) for k, v in compat.iteritems(var.copy())),**kwargs ) elif isinstance(var, (list, tuple)): #my change here max_elements = 20 ret = func(name, [varmap(func, f, context, name, **kwargs) for f in var[:max_elements]],**kwargs) else: ret = func(name, var, **kwargs) context.remove(objid) return ret ``` ---
**Author:** ![basepi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/basepi/32/53056_2.png) [@basepi](https://discuss.elastic.co/u/basepi)\ **Post date:** [July 21, 2022, 5:36pm UTC](https://discuss.elastic.co/t/python-client-apm-very-slow-when-local-list-variable-is-very-large/309935/9 "2022-07-21T17:36:52Z")
Ah, yes, I see it now. `varmap` is recursing to the bottom, and shortening on the way back up. That's definitely a problem. Can you please open an [issue](https://github.com/elastic/apm-agent-python/issues/new?assignees=&labels=&template=Bug_report.md) so we can get this fixed? ---
**Author:** ![Kohlman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kohlman/32/87348_2.png) [@Kohlman](https://discuss.elastic.co/u/Kohlman)\ **Post date:** [July 21, 2022, 8:16pm UTC](https://discuss.elastic.co/t/python-client-apm-very-slow-when-local-list-variable-is-very-large/309935/10 "2022-07-21T20:16:13Z")
> > > \*\*Describe the bug\*\*: ... > When python code throws an error, and one of the loca…l variables contains a very large list of lists of lists, it takes the elastic python apm client a very long time to process through the list. In our case it was taking over 60 seconds for the code to complete. > > > \*\*To Reproduce\*\* > To simply replicate the issue, I: (see code snipit below) > > created a simple script to create a list of list of lists of integers. > in one method I had my "with elasticAPMTransaction:" statement. > under this I created the list (200 items) of list (300 items) of list (40 items) aka 24M items total. > started to iterate through the outer list and picked an arbitrary spot to through an exception. This essentially mimicked the method where the real code threw an exception. > wrapped this whole thing in a profiler and evaluated the output. > This would take 60 seconds +/- to complete. > > Seems like the issue is in the varmap function in elasticapm\\utils\_init\_.py but there maybe lots of ways to solve this issue. > > \*\*Environment (please complete the following information)\*\* > \- OS: \[e.g. Linux\] > \- Python version: 3.x > \- Framework and version \[e.g. Django 2.1\]: > \- APM Server version: N/A > \- Agent version: 6.7.2 > > > \*\*Additional context\*\* > Created a case here https://discuss.elastic.co/t/python-client-apm-very-slow-when-local-list-variable-is-very-large/309935. See this for additional details. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 11, 2022, 4:16pm UTC](https://discuss.elastic.co/t/python-client-apm-very-slow-when-local-list-variable-is-very-large/309935/11 "2022-08-11T16:16:46Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.