Elastic Endpoint (Defend) does not seem to report file hashes for writes or modifications

Elastic Security Endpoint Security
1

Hey all,

this is similar to: Endpoint events dont contain process or file hash

When I look at fields in the endpoint* datasets, the fields file.hash.* are not populated. I see process.hash.* on events in the dataset endpoint.events.process, but no file hashes for file write / modify events.

This is a super common feature of any EDR that I know, so I was wondering if this could be a misconfiguration or bug, or if Elastic Endpoint simply can't report file hashes of files that were interacted with on the endpoint.

I don't know how to export my endpoint policy via API, but here are some details.

  • Elastic Agent Version: 8.14.1
  • Protected Endpoint OS: Windows Server 2019, Windows 10, Ubuntu Linux
  • Elastic Defend Integration Version: 8.15.2-prerelease.0
  • All Collectors enabled in Endpoint Policy, except "Collect Session Data" for Linux
  • no trusted applications or event filters configured
  • Advanced Settings mostly on default, except:
  • linux.advanced.events.enable_caps = true
  • ancestry depth set to 20
  • advanced.events.deduplicate_network_events = false
    (I don't know why these are set, I have not manually changed anything within the policies).
2

Hey @nemhods,

For performance reasons, Endpoint does not populate hashes for file events. Hash computation is a very I/O- and CPU- intensive action. Reading and hashing every every file modified on the system can significantly impact system responsiveness. This becomes especially noticeable when the system is performing heavy I/O, which is usually correlated with file event generation.

Malware alerts (event.code: malicious_file) contain hashes.

Regards,
Gabriel

1 Like
3

Hey Gabriel,

I appreciate it. Has there been any discussion to make this user-configurable?

It would really help with alerts like Potential Lateral Tool Transfer via SMB Share | Elastic Security Solution [8.14] | Elastic - a file has been dropped via the network, but I can't check this file in VirusTotal or other Threat Intel (neither with elastic's integrated TI features).

4

Would grabbing the file help? Endpoint response actions | Elastic Security Solution [8.15] | Elastic

You can automate it via the API too: Get a file from a host | Elastic Security Solution [8.15] | Elastic

5

Hey,

it's a good suggestion, but I wouldn't say that's an adequate workaround...

  • needs enterprise license
  • files are often ephemeral in a way, and may be gone or moved when the analyst tries to grab the file

I would love to be able to trade some performance for additional visibility. an advanced config option would be great!

6

Sure thing. I filed a feature request in our public Endpoint repo. Please feel free to comment on it. I can't guarantee that our engineering/product teams will prioritize it, but at least you can follow the issue to track progress.

Regards,
Gabriel

7

Hey Gabriel,

thanks for going ahead and adding the request.

Although I am wondering - if Elastic Agent doesn't ever compute the hashes of files, how can it enforce Blocklist entries? Do these only apply to executed files? This would mean that I couldn't effectively block the hash of a malicious .js-File, since the executed file in this case would be wscript.exe, not the .js-File.

Do you maybe have some insight why other solutions routinely have file hashes available? Do they use an entirely different architecture, or do they just accept the performance impact without giving the customer a choice about it?

Here's Microsoft's Defender data schema for File Events: DeviceFileEvents table in the advanced hunting schema - Microsoft Defender XDR | Microsoft Learn

8

That's correct. The blocklist is an extension of malware protection, which protects against malicious executables. On Windows, this means PE files.
Per our policy docs:

The blocklist allows you to prevent specified applications from running on hosts, extending the list of processes that Elastic Defend considers malicious.

The feature is actually a bit broader than that. Malware protection also protects against other PE files such as DLLs and driver (SYS) files.

Thank you. We'll take that into account when discussing this feature. It's important to understand that regardless of whether MDE spends CPU and I/O to hash files, adding them to Endpoint will dramatically increase its CPU and I/O usage in common heavy-system-load scenarios such as directory copies and compilation. Every file created must be re-read into memory then hashed, effectively doubling I/O and spiking CPU right when the system needs those resources. That means the user's foreground activity can be adversely affected, leading to upset [and potentially lost] customers. Making the feature opt-in can avoid these problems for most users, while providing the functionality for users who want it, which is why I'm requesting that approach.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.