Let's say I want to monitor the system logs of a cluster of servers with filebeat. Because using one data stream per host doesn't scale well and the cluster is logically part of the same application, they all write to the same data stream with minimal write-only privileges.
I'm struggling to figure out how to avoid two potential obstacles during post mortem analysis in the case of a compromised host:
- The lack of a server side timestamp (a "processed_at" in addition to the normal timestamp) allows a malicious client to date logs to an arbitrary time. While this doesn't allow overwriting existing documents, it makes it difficult to simply discard all events after a known-bad date.
- A malicious client could impersonate other clients by adding documents with spoofed identifier fields, making it more difficult to reconstruct the actual course of events.
The ingest pipeline includes a set_security_user processor that would solve the attribution problem, but to my understanding, the pipeline application cannot be enforced.
What are my options?
Thanks in advance