Hello everyone,
I am currently working with Elasticsearch v8.17.3 using the Custom Logs (Filestream) integration via Elastic Agent.
I have encountered a problem:
- Filestream only starts ingesting a file when it is larger than ~1 KB.
- I found in the documentation that this can be adjusted via
prospector.scanner.fingerprint.length(minimum 64 bytes), so I changed it to 64 bytes.
However, I now face two issues:
- The change doesn’t seem to work — even when my file is over 64 bytes, ingestion doesn’t start until it reaches around 1 KB. I’m confused about the exact syntax/IDs needed to set this correctly in the Elastic Agent integration policy (Fleet). Could someone provide a working example?
- My use case requires real-time ingestion of every new log line (even ~20 bytes), because I use these logs to trigger detection rules. Even 64 bytes is much larger than I need — ideally, ingestion should happen immediately when a new line is added.
Questions:
- Is there any way in v8.17.3 to bypass the 1 KB file-size threshold so Filestream starts reading immediately?
- If not, is there a supported workaround (e.g., file_identity mode change, using standalone Filebeat, or another method) that can ingest small logs in real time?
- Could someone share the exact policy configuration (with correct field names) to change the fingerprint length in Fleet-managed Elastic Agent?
This is urgent for me — my Master’s thesis depends on getting real-time log ingestion working. Any advice or alternative solution would be greatly appreciated.
Thanks in advance!