# Need to Ingest Small Log Files in Real-Time 

**URL:** <https://discuss.elastic.co/t/need-to-ingest-small-log-files-in-real-time/380826>\
**Category:** Elastic Agent\
**Created:** [August 6, 2025, 1:34pm UTC](https://discuss.elastic.co/t/need-to-ingest-small-log-files-in-real-time/380826 "2025-08-06T13:34:49Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nourgr21](https://avatars.discourse-cdn.com/v4/letter/n/46a35a/32.png) [@Nourgr21](https://discuss.elastic.co/u/Nourgr21)\
**Post date:** [August 6, 2025, 1:34pm UTC](https://discuss.elastic.co/t/need-to-ingest-small-log-files-in-real-time/380826/1 "2025-08-06T13:34:49Z")

</div>

Hello everyone,

I am currently working with **Elasticsearch v8.17.3** using the **Custom Logs (Filestream)** integration via Elastic Agent.

I have encountered a problem:

- Filestream **only starts ingesting a file when it is larger than ~1 KB**.
- I found in the documentation that this can be adjusted via `prospector.scanner.fingerprint.length` (minimum 64 bytes), so I changed it to **64 bytes**.

However, I now face **two issues** :

1. **The change doesn’t seem to work** — even when my file is over 64 bytes, ingestion doesn’t start until it reaches around 1 KB. I’m confused about the exact syntax/IDs needed to set this correctly in the Elastic Agent integration policy (Fleet). Could someone provide a working example?
2. My use case requires **real-time ingestion of every new log line** (even ~20 bytes), because I use these logs to trigger **detection rules**. Even 64 bytes is much larger than I need — ideally, ingestion should happen immediately when a new line is added.

**Questions:**

- Is there any way in v8.17.3 to bypass the 1 KB file-size threshold so Filestream starts reading immediately?
- If not, is there a supported workaround (e.g., file\_identity mode change, using standalone Filebeat, or another method) that can ingest small logs in real time?
- Could someone share the **exact policy configuration** (with correct field names) to change the fingerprint length in Fleet-managed Elastic Agent?

This is urgent for me — my **Master’s thesis depends on getting real-time log ingestion working**. Any advice or alternative solution would be greatly appreciated.

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 6, 2025, 1:40pm UTC](https://discuss.elastic.co/t/need-to-ingest-small-log-files-in-real-time/380826/2 "2025-08-06T13:40:57Z")

</div>

Closed as duplicate of

> [@Need to Ingest Small Log Files in Real-Time](https://discuss.elastic.co/t/need-to-ingest-small-log-files-in-real-time/380823):
>
> Hello everyone, I am currently working with Elasticsearch v8.17.3 using the Custom Logs (Filestream) integration via Elastic Agent. I have encountered a problem: Filestream only starts ingesting a file when it is larger than ~1 KB. I found in the documentation that this can be adjusted via prospector.scanner.fingerprint.length (minimum 64 bytes), so I changed it to 64 bytes. However, I now face two issues: The change doesn’t seem to work — even when my file is over 64 bytes, ingestion doe…

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 6, 2025, 1:41pm UTC](https://discuss.elastic.co/t/need-to-ingest-small-log-files-in-real-time/380826/3 "2025-08-06T13:41:02Z")

</div>