okay interesting, thanks for looking into this @nemo; I just pushed another commit in which the pointers are getting "nil"ed at correct code paths, mitigating the panics you observed. Whenever you get the chance please do have another of your excellent experiments (really do appreciate your help). That said, during this second look, I noticed that every time a process forks a child, we copy all the resolved dns names of the parent to the child process. As a result of this design choice, I consider the memory step ups as normal and trying to make this more lightweight, in terms of memory, may need some revisiting
The growth of this memory is still very abnormal...It ran out of memory and caused the machine to crash...
Hello @Panos_Koutsovasilis, is there a public issue tracking this?
We use Elastic Agent and are planning to upgrade from 8.12, I saw this post but could not find any public issue tracking this so it is not clear if this also affects the auditbeat process used by the Elastic Agent.
A memory leak that could lead to the machine crashing seems pretty critical, we skipped 8.11 because a similar issue with metricbeat.
thank you for checking this again @nemo, I think that the design choice of cloning all map entries during fork is what causing this enormous memory footprint that your charts are capturing and it isn't a memory leak, but eitherway it needs further investigation. So then let's make you capture your nice graphs and all your effort in a issue here. This will help with the logistics and will get the necessary attention from the auditbeat owner team
PS: thanks for the LRU proposal this needs we will definitely consider it
Hi @leandrojmp 
@nemo was kind enough to do some experiments for us but at this time point I am not 100% sure of the reason (if it is a memory leak) or a memory heavy design choice. I am leaning towards the latter. Either way, @nemo will capture this in an issue on the beats repo and the team will follow up with updates there
1 Like