# Filestream multiple file reading issue **URL:** \ **Category:** Elastic Agent\ **Created:** [June 4, 2025, 9:49am UTC](https://discuss.elastic.co/t/filestream-multiple-file-reading-issue/378847 "2025-06-04T09:49:21Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![BnB](https://avatars.discourse-cdn.com/v4/letter/b/da6949/32.png) [@BnB](https://discuss.elastic.co/u/BnB)\ **Post date:** [June 4, 2025, 9:49am UTC](https://discuss.elastic.co/t/filestream-multiple-file-reading-issue/378847/1 "2025-06-04T09:49:21Z")
So here's my case: Im using Custom Logs (Filestream) integration on Elasticsearch. The way the files I want it to read is as follows: 1. There are already multiple files that fit the "Paths" config, so it looks like this Paths: /path/to/files/logs\_to\_read-\*.ndjson /path/to/files/ directory: logs\_to\_read-one.ndjson logs\_to\_read-two.ndjson logs\_to\_read-three.ndjson 1. Whenever new data is to be ingested, what happens is a new file in the /path/to/files/ directory is created, so it looks like this: /path/to/files/ directory after new data appears: logs\_to\_read-one.ndjson logs\_to\_read-two.ndjson logs\_to\_read-three.ndjson logs\_to\_read-four.ndjson I want this filestream configuration to: 1. Read all files that are already in the /path/to/files/ directory 2. Read any new file that appears in the /path/to/files/ directory Current Parsers configuration: ```auto - ndjson: target: "" overwrite_keys: true expand_keys: true ``` Current behavior: 1. No file is being read by filebeat 2. There are no errors in filebeat logs, the agents status is Healthy I would really appreciate any help. ---
**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\ **Post date:** [June 4, 2025, 11:00am UTC](https://discuss.elastic.co/t/filestream-multiple-file-reading-issue/378847/2 "2025-06-04T11:00:38Z")
Hi @BnB, Welcome! Can you share your input configuration for Filebeat and which version you are using? Are you using teh [debug options](https://www.elastic.co/docs/reference/beats/filebeat/enable-filebeat-debugging) as well to get more information in addition to the logs? I would expect your filestream input configuration to look a little like this: ```auto filebeat.inputs: - type: filestream id: multi-filestream paths: - /path/to/files/logs_to_read-*.ndjson ``` Are you using the [prospector option](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-filestream#filebeat-input-filestream-options) as well to look for files? That may be needed too. Let us know! ---
**Author:** ![BnB](https://avatars.discourse-cdn.com/v4/letter/b/da6949/32.png) [@BnB](https://discuss.elastic.co/u/BnB)\ **Post date:** [June 4, 2025, 11:10am UTC](https://discuss.elastic.co/t/filestream-multiple-file-reading-issue/378847/3 "2025-06-04T11:10:22Z")
Hi, thank you for your time. Filestream version: v1.1.3 Input: /home/new/\*.ndjson In „Edit Custom Logs (Filestream) integration” there is not such option as debug option or prospector or at least I'm not able to locate them, sorry. ---
**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\ **Post date:** [June 4, 2025, 11:12am UTC](https://discuss.elastic.co/t/filestream-multiple-file-reading-issue/378847/4 "2025-06-04T11:12:18Z")
The details on how to configure filebeat with the debug level options are [here](https://www.elastic.co/docs/reference/beats/filebeat/enable-filebeat-debugging). Those are not specific to the prospector. Can you share any output you see with the debug option enabled? ---
**Author:** ![BnB](https://avatars.discourse-cdn.com/v4/letter/b/da6949/32.png) [@BnB](https://discuss.elastic.co/u/BnB)\ **Post date:** [June 4, 2025, 12:28pm UTC](https://discuss.elastic.co/t/filestream-multiple-file-reading-issue/378847/5 "2025-06-04T12:28:37Z")
here's the content of todays log file: > note that i deleted lines containing: CA certificate matching 'ca\_trusted\_fingerprint' found, adding it to 'certificate\_authorities ca\_trusted\_fingerprint' set, looking for matching fingerprints Non-zero metrics in the last 30s since there were a lot of them, and made the file so big it couldnt be uploaded, and didnt seem to be relevant ---
**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\ **Post date:** [June 5, 2025, 9:56am UTC](https://discuss.elastic.co/t/filestream-multiple-file-reading-issue/378847/6 "2025-06-05T09:56:01Z")
Thanks @BnB. Ah it's the [Agent Custom Filestream Log integration](https://www.elastic.co/guide/en/integrations/current/filestream.html) rather than Filebeat specifically. I've removed the filebeat tag to avoid confusion. I see some healthy messages on the filestream, and some fleet-server errors (Possible transient error during checkin with fleet-server) that should be recoverable. But it looks to be info level rather than debug so it may be worth changing the setting level. Can you please also share your full configuration? ---
**Author:** ![BnB](https://avatars.discourse-cdn.com/v4/letter/b/da6949/32.png) [@BnB](https://discuss.elastic.co/u/BnB)\ **Post date:** [June 5, 2025, 10:22am UTC](https://discuss.elastic.co/t/filestream-multiple-file-reading-issue/378847/7 "2025-06-05T10:22:16Z")
sorry for the confusion, here comes the configuration: > ---
**Author:** ![BnB](https://avatars.discourse-cdn.com/v4/letter/b/da6949/32.png) [@BnB](https://discuss.elastic.co/u/BnB)\ **Post date:** [June 10, 2025, 8:08am UTC](https://discuss.elastic.co/t/filestream-multiple-file-reading-issue/378847/8 "2025-06-10T08:08:49Z")
![image](https://us1.discourse-cdn.com/elastic/optimized/3X/a/8/a8d2818abadc61df5c115e6d972ccd0b8e1814ad_2_690x21.png) I have noticed in my logs that a harvester is started for paths "/var/log/messages\* /var/log/syslog\* /var/log/system\*" and not the path I specified in configuration, is this normal behavior? ---
**Author:** ![BnB](https://avatars.discourse-cdn.com/v4/letter/b/da6949/32.png) [@BnB](https://discuss.elastic.co/u/BnB)\ **Post date:** [June 11, 2025, 12:00pm UTC](https://discuss.elastic.co/t/filestream-multiple-file-reading-issue/378847/9 "2025-06-11T12:00:23Z")
hi, sorry for bothering you again > could these be more helpful/useful? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [June 11, 2025, 1:14pm UTC](https://discuss.elastic.co/t/filestream-multiple-file-reading-issue/378847/10 "2025-06-11T13:14:03Z")
VAR syslog above is most likely because you have the system integration enabled as well. Those are the paths that it will read. With respect to your configuration, try taking out the json parser it could be failing and you're getting no logs. Otherwise it looks pretty good at a glance. Are you using the UI to configure this or are you purely doing a self-managed agent with this configuration? Also, I'm not sure if that path to the files is the actual path or just a sample, but either way you need to make sure that that entire directory tree and files are readable. We often run into folks that the actual file is readable but the parent directories are not so make sure you check that as well Also as Carly said the debug logs would help not just the info If you're using the UI, perhaps a couple screenshots ---
**Author:** ![BnB](https://avatars.discourse-cdn.com/v4/letter/b/da6949/32.png) [@BnB](https://discuss.elastic.co/u/BnB)\ **Post date:** [June 12, 2025, 9:36am UTC](https://discuss.elastic.co/t/filestream-multiple-file-reading-issue/378847/11 "2025-06-12T09:36:59Z")
Hi, thanks for the reply. 1. deletion of json parser didn't help sadly 2. the path is a sample, i will check the directory readabilities once i get access to the server wich should be soon enough 3. Unfortunately these are the only "debug" logs I found in the agent, i found therese something as Log4j thats used for logging in elastic but I'll need a while to configure it so if Log4j is able to help Ill come back once its up and running ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/a/0ad7abe88bf5a215dcea3a1085a9114611437bb1.png) ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [June 12, 2025, 4:01pm UTC](https://discuss.elastic.co/t/filestream-multiple-file-reading-issue/378847/12 "2025-06-12T16:01:51Z")
To change the logging level ..... Go to the agent .... And set the log level... Yeah not obvious ![Screenshot 2025-06-12 at 8.57.36 AM](https://us1.discourse-cdn.com/elastic/original/3X/2/3/23e4936ec4396095fd98eefa97a5d4c22d89a776.png) Also to get a better view of the logs Go To Discover Data View : `logs-*` KQL Bar: `data_stream.dataset : "elastic_agent.filebeat"` Add the `message` and `log.level` fields in the display ![Screenshot 2025-06-12 at 9.01.08 AM](https://us1.discourse-cdn.com/elastic/original/3X/6/d/6d38241d5cf8e36d7c479509fad894c76191c72d.jpeg) ---
**Author:** ![BnB](https://avatars.discourse-cdn.com/v4/letter/b/da6949/32.png) [@BnB](https://discuss.elastic.co/u/BnB)\ **Post date:** [June 13, 2025, 6:52am UTC](https://discuss.elastic.co/t/filestream-multiple-file-reading-issue/378847/13 "2025-06-13T06:52:42Z")
Okay, I have done that and the logs it started to show me are: cannot start ingesting from file "path/to/file/file.ndjson": filesize of "path/to/file/file.ndjson" is 99 bytes, expected at least 1024 bytes for fingerprinting: file size is too small for ingestion Unfortunately niether turning fingerprinting off nor lowering the fingerprint length helped. ---
**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\ **Post date:** [June 13, 2025, 9:34am UTC](https://discuss.elastic.co/t/filestream-multiple-file-reading-issue/378847/14 "2025-06-13T09:34:19Z")
Thanks for sharing @BnB. > [@BnB](#): > > cannot start ingesting from file "path/to/file/file.ndjson": filesize of "path/to/file/file.ndjson" is 99 bytes, expected at least 1024 bytes for fingerprinting: file size is too small for ingestion The issue is down to an existing filestream limitation where the file needs to be at least 1kB. This is because the default file identity has changed from native to fingerprint in v9, [as covered in the release notes](https://www.elastic.co/docs/release-notes/beats/breaking-changes#beats-900-breaking-changes). There is already an issue [here](https://github.com/elastic/beats/issues/44780) discussing possible approaches to address this limitation. There is a way to revert to the 8.x behaviour in the release notes: > To preserve the behaviour from 8.x, set `file_identity.native: ~` and `prospector.scanner.fingerprint.enabled: false` Can you try setting those options and see if the file is now picked up? Let us know! ---
**Author:** ![BnB](https://avatars.discourse-cdn.com/v4/letter/b/da6949/32.png) [@BnB](https://discuss.elastic.co/u/BnB)\ **Post date:** [June 17, 2025, 9:03am UTC](https://discuss.elastic.co/t/filestream-multiple-file-reading-issue/378847/15 "2025-06-17T09:03:49Z")
Hi, I've been digging around in the options but i can't find a way to add "file\_identity.native: ~" to my configuration, I've looked for the option in GUI but found nothing, ive looked around for the raw .yml file in servers filesystem, andtrued to modify the PUT request made by GUI but none of these offered a way to add this one option. ![obraz (9)](https://us1.discourse-cdn.com/elastic/original/3X/5/3/53f5705ef6ee0ae9db3b16af925cefe7c396b5a5.png) ---
**Author:** ![BnB](https://avatars.discourse-cdn.com/v4/letter/b/da6949/32.png) [@BnB](https://discuss.elastic.co/u/BnB)\ **Post date:** [June 24, 2025, 12:39pm UTC](https://discuss.elastic.co/t/filestream-multiple-file-reading-issue/378847/16 "2025-06-24T12:39:59Z")
So since last time I've dug around and found out two things: 1. The [filestream documentation](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-filestream#_file_identity_2) deems fingerprint to be the default file identity implementation. 2. The Custom Logs (Filestream) API reference doesn't mention any other file identity implementation except for fingerprint. So the other file identity options exist but they are not accessible through the Elastic Agents API, while turning fingerpring off is an option it probably does not affect anything since it's the default option (I assume that if no file identity implementation is chosen Elastic just uses the default). Additionally the filestream documentation mentions that " Changing `file_identity` is only supported from `native` or `path` to `fingerprint`" so if I had an option to select the file identity I want id need to create the Elastic Agent from scratch. Since I dont want to create a standalone Filestream agent it seems my only options of resolving this issue are: 1. Filling my files with filler bytes 2. Waiting for the Agents API to get uptdated with the missing file identity options 3. Waiting for the fingerprint file identity implementation to get updated so it can work with files smaller than 1024 bytes For those who come after