# Filebeat does not detect log files (9.0.0) **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [April 30, 2025, 10:02am UTC](https://discuss.elastic.co/t/filebeat-does-not-detect-log-files-9-0-0/377662 "2025-04-30T10:02:42Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![lbrame-work](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lbrame-work/32/142895_2.png) [@lbrame-work](https://discuss.elastic.co/u/lbrame-work)\ **Post date:** [April 30, 2025, 10:02am UTC](https://discuss.elastic.co/t/filebeat-does-not-detect-log-files-9-0-0/377662/1 "2025-04-30T10:02:42Z")
Hello! First of all, I hope this is the right place to ask for help. I have been trying to use `filebeat` to forward application logs generated by an API application to a `logstash` instance, which then applies the necessary transformations to forward the data to Elasticsearch. The problem I am facing is that `filebeat` does not seem to be picking up the log files that it is pointed to. ## Setup On this machine, I am running Windows 11 Pro. I have tried to run `filebeat` both within a WSL instance (Fedora 42 WSL) and natively, on Windows, using the x86\_64 zip. In both environments, I have tried to run the program both as a regular user and as an elevated one (root / Administrator), starting with a fresh setup each time. On top of that, in order to avoid permissions issues, the `logs` folder that I am trying to read is present both on the native NTFS filesystem, and within the WSL filesystem, and I have only tried to use the correct one for each instance of Filebeat running. ## Symptom Filebeat does not seem to be logging the files. Latest log files in `./logs/` do not indicate the harvester has picked up any files: ```auto // ... "filebeat":{"harvester":{"open_files":0,"running":0}} // ... ``` Also, all the files in `./registry/filebeat/*` do not have any signs of the log files from the API. The only files that show up there are some Linux log files, in the WSL instance, when I add `/var/log/*.log` to the `paths` list in the `filestream` input, ## Configuration I am using the following `filebeat.yml`: ```auto filebeat.inputs: - type: filestream id: test-filestream-id enabled: true paths: - C:\Users\luca\RiderProjects\MinimalAPITest\MinimalAPITest\logs\*.log filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false output.logstash: enabled: true hosts: ["logstash-instance-hostname:5044"] ``` Noting that, when running via WSL, the Windows path is replaced by a path to the same directory copied within the Linux filesystem: ```auto /home/luca/RiderProjects/MinimalAPITest/MinimalAPITest/logs/*.log ``` The `logs` directory only contains one `testapi.log` file, generated from a dotnet application. Of course, I am not able to find the data on the Elastic instance. I strongly suspect that is because there is something wrong with my configuration, so, no data is sent out. `./filebeat test config` and `./filebeat test output` both return a successful output. Thank you in advance, any help is really appreciated ๐Ÿ™‚ ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 30, 2025, 8:00pm UTC](https://discuss.elastic.co/t/filebeat-does-not-detect-log-files-9-0-0/377662/2 "2025-04-30T20:00:08Z")
Hi @lbrame-work Welcome to community. Perhaps take a look here > **[Avoid YAML formatting problems | Elastic Documentation](https://www.elastic.co/docs/reference/beats/filebeat/yaml-tips#wrap-paths-in-quotes)** > > The configuration file uses YAML for its syntax. When you edit the file to modify configuration settings, there are a few things that you should know... > Wrap paths in single quotation marks > Windows paths in particular sometimes contain spaces or characters, such as drive letters or triple dots, that may be misinterpreted by the YAML parser. > To avoid this problem, itโ€™s a good idea to wrap paths in single quotation marks. ---
**Author:** ![lbrame-work](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lbrame-work/32/142895_2.png) [@lbrame-work](https://discuss.elastic.co/u/lbrame-work)\ **Post date:** [May 5, 2025, 7:52am UTC](https://discuss.elastic.co/t/filebeat-does-not-detect-log-files-9-0-0/377662/3 "2025-05-05T07:52:27Z")
Hello! Thank you for your advice. I have tried applying the rules cited here but, sadly, that did not seem to make a difference. During this weekend, I have even attempted to reproduce this on my personal machine, which runs a on-the-metal Fedora 42 install, with no avail. To rule out SELinux-derived side effects, I also decided to try the same thing on a fresh Ubuntu VM, but no changes still. The relevant part of my `filebeat.yml` now looks like this: ```yaml filebeat.inputs: - type: filestream id: test-fiestream-id enabled: true paths: #- '/var/log/*.log' - '/home/luca/RiderProjects/MinimalAPITest/MinimalAPITest/logs/*.log' ``` Sadly, though, this does not seem to make a difference. `filebeat -d publish` still does not output anything. More so, when going without `/var/log`, the registry `log.json` file is still empty: ```zsh (2) ~/bin/filebeat-9.0.0-linux-x86_64/data/registry/filebeat % less log.json (2) ~/bin/filebeat-9.0.0-linux-x86_64/data/registry/filebeat % stat log.json File: log.json Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: 8,48 Inode: 83737 Links: 1 Access: (0600/-rw-------) Uid: ( 1000/ luca) Gid: ( 1000/ luca) Access: 2025-05-05 09:23:07.025304115 +0200 Modify: 2025-05-05 09:22:38.829317794 +0200 Change: 2025-05-05 09:22:38.829317794 +0200 Birth: 2025-05-05 09:22:38.829317794 +0200 ``` ## Facts about the log file and things I tried The log file which is being ignored, which I called `testapi.log`, is logged from a dotnet application using a Serilog file sink. This is the content of the file: ```auto 2025-05-05 09:28:29.230 +02:00 [INF] Hello, Serilog! ``` It has permissions `644` under Linux, which would be readable to `filebeat`. I have even tried setting its mode to `755`, but it makes no difference: ```auto % stat testapi.log File: testapi.log Size: 53 Blocks: 8 IO Block: 4096 regular file Device: 8,48 Inode: 89036 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 1000/ luca) Gid: ( 1000/ luca) Access: 2025-05-05 09:28:45.658658587 +0200 Modify: 2025-05-05 09:28:29.278668645 +0200 Change: 2025-05-05 09:28:29.278668645 +0200 Birth: 2025-05-05 09:28:29.198668694 +0200 ``` ## Copied files from `/var/log` get detected I have also found that files copied directly from `/var/log` into the `logs` folder work and they get copied. The same happens if I copy a file from `/var/log` into a fresh file I created myself: ```bash cat /var/log/dnf5.log > copied.log ``` The result is that both `dnf.log` and `copied.log` get caught by filebeat. It seems to be, indeed, about the **content** of the file. Removing all the logs and copying the contents in `/var/log/dnf5.log` into `testapi.log` also gets filebeat to see it. What I am not getting is that both the `testapi.log` file generated by Serilog and the `copied.log` file, copied from the contents of `/var/log/dnf5.log`, have the same permission and ownserhip: ```bash % /usr/bin/ls -lah total 228K drwxr-xr-x 2 luca luca 4.0K May 5 10:22 . drwxr-xr-x 6 luca luca 4.0K Apr 28 13:44 .. -rw-r--r-- 1 luca luca 215K May 5 10:22 copied.log -rw-r--r-- 1 luca luca 53 May 5 10:22 testapi.log ``` Is there any way Filebeat expects the log files to be formatted? Maybe the timestamp format is the culprit? Thanks again for the help! ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 5, 2025, 5:08pm UTC](https://discuss.elastic.co/t/filebeat-does-not-detect-log-files-9-0-0/377662/4 "2025-05-05T17:08:57Z")
Apologies I have kinda lost track... > [@lbrame-work](#): > > Is there any way Filebeat expects the log files to be formatted? Maybe the timestamp format is the culprit? Filebeat expects the logs to be newline delimited that is about it ... if it is a single line with no new line it will not process that line. > [@lbrame-work](#): > > `~/bin/filebeat-9.0.0-linux-x86_64/data/registry/filebeat` That is a weird place for the registry how did you install filebeat? > [@lbrame-work](#): > > ` - '/home/luca/RiderProjects/MinimalAPITest/MinimalAPITest/logs/*.log'` Do you mean to have the `MinimalAPITest` in there twice With respect to permessions ... I am sure you know but it is not just the file permissions the parent directories need to be readable as well... so does the user running filebeat have permission to ready from /var/log... Something simple going on... ---
**Author:** ![lbrame-work](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lbrame-work/32/142895_2.png) [@lbrame-work](https://discuss.elastic.co/u/lbrame-work)\ **Post date:** [May 6, 2025, 8:15am UTC](https://discuss.elastic.co/t/filebeat-does-not-detect-log-files-9-0-0/377662/5 "2025-05-06T08:15:51Z")
No worries. > [@stephenb](#): > > Filebeat expects the logs to be newline delimited that is about it ... if it is a single line with no new line it will not process that line. Thank you! I am now making sure that the test application prints at least 5 lines of logs, delimited by a newline. Unfortunately, though, this does not seem to make a difference. > [@stephenb](#): > > That is a weird place for the registry how did you install filebeat? I have installed Filebeat from the officially-provided tar archive, `./data` is where what otherwise would go in `/var/lib/filebeat` seems to go. > [@stephenb](#): > > Do you mean to have the `MinimalAPITest` in there twice Yes, that is correct. I have a Solution and a Project with the same name. > [@stephenb](#): > > With respect to permessions ... I am sure you know but it is not just the file permissions the parent directories need to be readable as well... so does the user running filebeat have permission to ready from /var/log... Yes, I have indeed checked the permissions as well. The users `filebeat` is running under (when running as `root`, without a doubt) have the permission to read all the directories I am making it read. It has no problem reading `/var/log` actually, the problem seems to be a particular file within the homedir. Directory mode is 755, and files that were copied from `/var/log` to this `/home/luca/RiderProjects/MinimalAPITest/MinimalAPITest/logs` are detected and logged, which means the directory permissions are okay. Both `/var/log` and the `logs` directory have `0755` access mode. * * * In the spirit of further troubleshooting, I have now switched from the .tar.gz archive to the native RPM, running Filebeat as root. This is the contents of my `testapi.log` file: ```auto 2025-05-06 09:46:59.994 +02:00 [INF] Hello, Serilog 2025-05-06 09:47:00.185 +02:00 [INF] Some new logs 2025-05-06 09:47:00.186 +02:00 [INF] Test 2025-05-06 09:47:00.186 +02:00 [INF] Something happened. ``` Running the system installation of filebeat from root,m the log not getting picked up, and it does not appear to be in `/var/lib/filebeat/registry/filebeat/log.json`. Also, I agree. I am sure something very simple must be going on, but I have been unable to figure it out thus far.