Filebeat does not detect log files (9.0.0)

[lbrame-work (Luca)]

Hello! First of all, I hope this is the right place to ask for help.

I have been trying to use filebeat to forward application logs generated by an API application to a logstash instance, which then applies the necessary transformations to forward the data to Elasticsearch.

The problem I am facing is that filebeat does not seem to be picking up the log files that it is pointed to.

Setup

On this machine, I am running Windows 11 Pro. I have tried to run filebeat both within a WSL instance (Fedora 42 WSL) and natively, on Windows, using the x86_64 zip. In both environments, I have tried to run the program both as a regular user and as an elevated one (root / Administrator), starting with a fresh setup each time. On top of that, in order to avoid permissions issues, the logs folder that I am trying to read is present both on the native NTFS filesystem, and within the WSL filesystem, and I have only tried to use the correct one for each instance of Filebeat running.

Symptom

Filebeat does not seem to be logging the files.

Latest log files in ./logs/ do not indicate the harvester has picked up any files:

// ...
"filebeat":{"harvester":{"open_files":0,"running":0}}
// ...

Also, all the files in ./registry/filebeat/* do not have any signs of the log files from the API. The only files that show up there are some Linux log files, in the WSL instance, when I add /var/log/*.log to the paths list in the filestream input,

Configuration

I am using the following filebeat.yml:

filebeat.inputs:
  - type: filestream
    id: test-filestream-id
    enabled: true
    paths:
      - C:\Users\luca\RiderProjects\MinimalAPITest\MinimalAPITest\logs\*.log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

output.logstash:
  enabled: true
  hosts: ["logstash-instance-hostname:5044"]

Noting that, when running via WSL, the Windows path is replaced by a path to the same directory copied within the Linux filesystem:

/home/luca/RiderProjects/MinimalAPITest/MinimalAPITest/logs/*.log

The logs directory only contains one testapi.log file, generated from a dotnet application.

Of course, I am not able to find the data on the Elastic instance. I strongly suspect that is because there is something wrong with my configuration, so, no data is sent out.

./filebeat test config and ./filebeat test output both return a successful output.

Thank you in advance, any help is really appreciated

[stephenb (Stephen Brown)]

Hi @lbrame-work Welcome to community.

Perhaps take a look here

Avoid YAML formatting problems | Elastic Documentation

Wrap paths in single quotation marks
Windows paths in particular sometimes contain spaces or characters, such as drive letters or triple dots, that may be misinterpreted by the YAML parser.

To avoid this problem, it’s a good idea to wrap paths in single quotation marks.

[lbrame-work (Luca)]

Hello! Thank you for your advice.

I have tried applying the rules cited here but, sadly, that did not seem to make a difference.

During this weekend, I have even attempted to reproduce this on my personal machine, which runs a on-the-metal Fedora 42 install, with no avail. To rule out SELinux-derived side effects, I also decided to try the same thing on a fresh Ubuntu VM, but no changes still.

The relevant part of my filebeat.yml now looks like this:

filebeat.inputs:
  - type: filestream
    id: test-fiestream-id
    enabled: true
    paths:
      #- '/var/log/*.log'
      - '/home/luca/RiderProjects/MinimalAPITest/MinimalAPITest/logs/*.log'

Sadly, though, this does not seem to make a difference. filebeat -d publish still does not output anything. More so, when going without /var/log, the registry log.json file is still empty:

(2) ~/bin/filebeat-9.0.0-linux-x86_64/data/registry/filebeat
% less log.json

(2) ~/bin/filebeat-9.0.0-linux-x86_64/data/registry/filebeat
% stat log.json
  File: log.json
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 8,48    Inode: 83737       Links: 1
Access: (0600/-rw-------)  Uid: ( 1000/    luca)   Gid: ( 1000/    luca)
Access: 2025-05-05 09:23:07.025304115 +0200
Modify: 2025-05-05 09:22:38.829317794 +0200
Change: 2025-05-05 09:22:38.829317794 +0200
 Birth: 2025-05-05 09:22:38.829317794 +0200

Facts about the log file and things I tried

The log file which is being ignored, which I called testapi.log, is logged from a dotnet application using a Serilog file sink. This is the content of the file:

2025-05-05 09:28:29.230 +02:00 [INF] Hello, Serilog!

It has permissions 644 under Linux, which would be readable to filebeat. I have even tried setting its mode to 755, but it makes no difference:

% stat testapi.log
  File: testapi.log
  Size: 53              Blocks: 8          IO Block: 4096   regular file
Device: 8,48    Inode: 89036       Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/    luca)   Gid: ( 1000/    luca)
Access: 2025-05-05 09:28:45.658658587 +0200
Modify: 2025-05-05 09:28:29.278668645 +0200
Change: 2025-05-05 09:28:29.278668645 +0200
 Birth: 2025-05-05 09:28:29.198668694 +0200

Copied files from /var/log get detected

I have also found that files copied directly from /var/log into the logs folder work and they get copied. The same happens if I copy a file from /var/log into a fresh file I created myself:

cat /var/log/dnf5.log > copied.log

The result is that both dnf.log and copied.log get caught by filebeat.

It seems to be, indeed, about the content of the file. Removing all the logs and copying the contents in /var/log/dnf5.log into testapi.log also gets filebeat to see it.

What I am not getting is that both the testapi.log file generated by Serilog and the copied.log file, copied from the contents of /var/log/dnf5.log, have the same permission and ownserhip:

% /usr/bin/ls -lah
total 228K
drwxr-xr-x 2 luca luca 4.0K May  5 10:22 .
drwxr-xr-x 6 luca luca 4.0K Apr 28 13:44 ..
-rw-r--r-- 1 luca luca 215K May  5 10:22 copied.log
-rw-r--r-- 1 luca luca   53 May  5 10:22 testapi.log

Is there any way Filebeat expects the log files to be formatted? Maybe the timestamp format is the culprit?

Thanks again for the help!

[stephenb (Stephen Brown)]

Apologies I have kinda lost track...

@lbrame-work

Is there any way Filebeat expects the log files to be formatted? Maybe the timestamp format is the culprit?

Filebeat expects the logs to be newline delimited that is about it ... if it is a single line with no new line it will not process that line.

@lbrame-work

~/bin/filebeat-9.0.0-linux-x86_64/data/registry/filebeat

That is a weird place for the registry how did you install filebeat?

@lbrame-work

- '/home/luca/RiderProjects/MinimalAPITest/MinimalAPITest/logs/*.log'

Do you mean to have the MinimalAPITest in there twice

With respect to permessions ... I am sure you know but it is not just the file permissions the parent directories need to be readable as well... so does the user running filebeat have permission to ready from /var/log...

Something simple going on...

[lbrame-work (Luca)]

No worries.

@stephenb

Filebeat expects the logs to be newline delimited that is about it ... if it is a single line with no new line it will not process that line.

Thank you! I am now making sure that the test application prints at least 5 lines of logs, delimited by a newline. Unfortunately, though, this does not seem to make a difference.

@stephenb

That is a weird place for the registry how did you install filebeat?

I have installed Filebeat from the officially-provided tar archive, ./data is where what otherwise would go in /var/lib/filebeat seems to go.

@stephenb

Do you mean to have the MinimalAPITest in there twice

Yes, that is correct. I have a Solution and a Project with the same name.

@stephenb

With respect to permessions ... I am sure you know but it is not just the file permissions the parent directories need to be readable as well... so does the user running filebeat have permission to ready from /var/log...

Yes, I have indeed checked the permissions as well. The users filebeat is running under (when running as root, without a doubt) have the permission to read all the directories I am making it read. It has no problem reading /var/log actually, the problem seems to be a particular file within the homedir. Directory mode is 755, and files that were copied from /var/log to this /home/luca/RiderProjects/MinimalAPITest/MinimalAPITest/logs are detected and logged, which means the directory permissions are okay. Both /var/log and the logs directory have 0755 access mode.

---

In the spirit of further troubleshooting, I have now switched from the .tar.gz archive to the native RPM, running Filebeat as root.

This is the contents of my testapi.log file:

2025-05-06 09:46:59.994 +02:00 [INF] Hello, Serilog
2025-05-06 09:47:00.185 +02:00 [INF] Some new logs
2025-05-06 09:47:00.186 +02:00 [INF] Test
2025-05-06 09:47:00.186 +02:00 [INF] Something happened.

Running the system installation of filebeat from root,m the log not getting picked up, and it does not appear to be in /var/lib/filebeat/registry/filebeat/log.json.

Also, I agree. I am sure something very simple must be going on, but I have been unable to figure it out thus far.