Elastic Endpoint Security crashes and memory errors

Elastic Security Endpoint Security
1

Hey,

I'm getting multiple different Elastic Endpoint crashes across different hosts and across Elastic Agent 7.14 and 7.13.2. I provide exemplary crashes below, from the Windows Application Event log.
In terms of crash reasons, i get:

  • 0xC00000F (STATUS_STACK_OVERFLOW, A new guard page for the stack cannot be created.)
  • 0xc0000005 (STATUS_ACCESS_VIOLATION)
  • 0xc0000409 (STATUS_STACK_BUFFER_OVERRUN, The system detected an overrun of a stack-based buffer in this application. This overrun could potentially allow a malicious user to gain control of this application.)
Name der fehlerhaften Anwendung: elastic-endpoint.exe, Version: 7.14.0.0, Zeitstempel: 0x610036d7
Name des fehlerhaften Moduls: elastic-endpoint.exe, Version: 7.14.0.0, Zeitstempel: 0x610036d7
Ausnahmecode: 0xc0000005
Fehleroffset: 0x00000000000ea382
ID des fehlerhaften Prozesses: 0xd0c
Startzeit der fehlerhaften Anwendung: 0x01d793a22cc2b1d0
Pfad der fehlerhaften Anwendung: C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe
Pfad des fehlerhaften Moduls: C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe
Berichtskennung: 813d2b33-4cb2-49b1-ada6-c86db832857a
Vollständiger Name des fehlerhaften Pakets: 
Anwendungs-ID, die relativ zum fehlerhaften Paket ist: 

Name der fehlerhaften Anwendung: elastic-endpoint.exe, Version: 7.14.0.0, Zeitstempel: 0x610036d7
Name des fehlerhaften Moduls: elastic-endpoint.exe, Version: 7.14.0.0, Zeitstempel: 0x610036d7
Ausnahmecode: 0xc0000409
Fehleroffset: 0x000000000019e8ed
ID des fehlerhaften Prozesses: 0x2654
Startzeit der fehlerhaften Anwendung: 0x01d794bf8f0635dc
Pfad der fehlerhaften Anwendung: C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe
Pfad des fehlerhaften Moduls: C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe
Berichtskennung: dcd5430a-9476-49b8-8567-2b6586da6975
Vollständiger Name des fehlerhaften Pakets: 
Anwendungs-ID, die relativ zum fehlerhaften Paket ist: 

Name der fehlerhaften Anwendung: elastic-endpoint.exe, Version: 7.13.2.0, Zeitstempel: 0x60c23327
Name des fehlerhaften Moduls: elastic-endpoint.exe, Version: 7.13.2.0, Zeitstempel: 0x60c23327
Ausnahmecode: 0xc0000005
Fehleroffset: 0x00000000000daac2
ID des fehlerhaften Prozesses: 0xd50
Startzeit der fehlerhaften Anwendung: 0x01d794e2dac13b47
Pfad der fehlerhaften Anwendung: C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe
Pfad des fehlerhaften Moduls: C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe
Berichtskennung: 3b6c885a-15f5-456c-87c6-7a27a4f82dea
Vollständiger Name des fehlerhaften Pakets: 
Anwendungs-ID, die relativ zum fehlerhaften Paket ist:

I'll try to set up a system to take automatic crash dumps, so maybe I can provide a .dmp file.

2

Hi @nemhods,

Thank you for the bug report. Two of those WER reports may have given us a general idea of the location of a crash, but one of them isn't very useful. Full crash dumps will provide much more context. We actually enable automatic crash dumps for our own process. Could you please collect any dumps from C:\Program Files\Elastic\Endpoint\cache\CrashDumps, zip them, and share them on our secure upload service? The directory is locked down, so you may need to first copy them elsewhere from an Administrative prompt.

This write-only upload link is specific to your case, and will only be accessible by our team: Elastic Upload Service : Upload

Regards,
Gabriel

3

Hey @gabriel.landau,

I managed to fetch one crash dump from a STATUS_ACCESS_VIOLATION issue. It's uploaded now. As for the other hosts: two are currently offline (they are client PCs) and one system has had its Fleet Policy changed, and the Elastic Endpoint directory was removed in the process. This was the system with the STATUS_STACK_BUFFER_OVERRUN issue, so if that comes up again, I will upload this crashdump too.

I hope this already helps somewhat!

4

Thank you @nemhods. Files received. The team is looking at it. It's late in the day and some people are out tomorrow (Friday). We may not have a response until tomorrow or Monday.

Regards,
Gabriel

5

No problem. I should add, the Agent resumes working after the crashes, so its not urgent from my point of view.

6

Thanks. We've hit a hiccup. Because we are a Protected Process, Windows Error Reporting (WER) is encrypting our dumps asymmetrically, in a way that only Microsoft can decrypt. They realized the headache that this behavior created for AntiMalware vendors, so they reversed this behavior in 20H1, but older versions of Windows are still affected. To demonstrate, here's your dump against an unrelated dump:

C:\Users\user\Downloads>file python.DMP elastic-endpoint.exe.3268.dmp
python.DMP:                    Mini DuMP crash report, 11 streams, Thu Jun 24 21:23:36 2021, 0x1806 type
elastic-endpoint.exe.3268.dmp: data

We have a second crash dump mechanism that may have produced data. It doesn't use WER, so the aforementioned encryption is not a concern. Unfortunately it doesn't fire for every type of crash, but there may be data here. Can you check for C:\Program Files\Elastic\Endpoint\cache\elasticendpoint.dmp?

7

Windows Error Reporting (WER) is encrypting our dumps asymmetrically

I was wondering why zipping didn't decrease file size...

You're in luck it seems, the crash dump is there and also now uploaded. Let me know if theres anything else I can assist with.