#edited the /lib/systemd/system/filebeat.target file
added the User=filebeat under the [Service] stanza
#Added some facl's for access for the filebeat user:
###read access for the log files specified in the prospector
setfacl -m u:filebeat:r /var/log/mylogfile
###read access for the filebeat.yml
setfacl -m u:filebeat:r /etc/filebeat/filebeat.yml
###execute access for filebeat executable
setfacl -m u:filebeat:x /usr/bin/filebeat
###write access for the filebeat log file
setfacl -m u:filebeat:w /var/log/filebeat/filebeat.log
###read/write access for the registry file
setfacl -m u:filebeat:rw /var/lib/filebeat/registry
However when I do a systemctl daemon-reload and restart the filebeat service it is stuck and not actively prospecting the log file. Is there a permission that I missed that is needed for proper functioning of filebeat?
I already have filebeat set to log in debug mode in my filebeat.yml file. This made no difference as the output is the same:
[root@myvm ~]# service filebeat start -d '*'
Starting filebeat (via systemctl): [ OK ]
The last log entry I have is:
2016-08-25T10:15:44-04:00 INFO Cleaning up filebeat before shutting down.
Nothing further, no acknowledgement of the process starting even though it has:
[root@myvm ~]# systemctl status filebeat.service
● filebeat.service - filebeat
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; disabled; vendor preset: disabled)
Active: active (running) since Thu 2016-08-25 10:24:17 EDT; 2h 27min ago
Docs: Filebeat Reference [8.11] | Elastic
Main PID: 5627 (filebeat)
CGroup: /system.slice/filebeat.service
└─5627 /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
Aug 25 12:51:35 myvm /usr/bin/filebeat[5627]: prospector.go:185: Start next scan
Aug 25 12:51:35 myvm /usr/bin/filebeat[5627]: prospector.go:261: scan path /var/log/mylogfile
Aug 25 12:51:40 myvm /usr/bin/filebeat[5627]: spooler.go:97: Flushing spooler because of tim...: 0
Aug 25 12:51:45 myvm /usr/bin/filebeat[5627]: prospector.go:185: Start next scan
Aug 25 12:51:45 myvm /usr/bin/filebeat[5627]: prospector.go:261: scan path /var/log/mylogfile
Aug 25 12:51:47 myvm /usr/bin/filebeat[5627]: spooler.go:97: Flushing spooler because of tim...: 0
Aug 25 12:51:55 myvm /usr/bin/filebeat[5627]: spooler.go:97: Flushing spooler because of tim...: 0
Aug 25 12:51:55 myvm /usr/bin/filebeat[5627]: prospector.go:185: Start next scan
Aug 25 12:51:55 myvm /usr/bin/filebeat[5627]: prospector.go:261: scan path /var/log/mylogfile
Aug 25 12:52:02 myvm /usr/bin/filebeat[5627]: spooler.go:97: Flushing spooler because of tim...: 0
Is it working and just not writing output to the log file? I would suspect it still isn't working because prompting changes to the file does not prompt changes in my output plugin.
If you mean in the filebeat configuration, I already have that set by having the logging level set to debug (excuse the indenting; it isn't copying over well but I assure you it is correct in my configuration):
Getting slightly closer to an answer here. Instead of facl's I changed ownership of the /var/lib/filebeat and /etc/filebeat and /var/log/filebeat/ directories to my filebeat user. Oh and I took out the runoptions, it got a little confusing when it kept causing some sort of log file rotation. Now I am getting output but nothing is presumably being scanned? Just a continuous loop of the following three lines, even when I change the contents of the log file:
2016-08-25T15:18:19-04:00 DBG Start next scan
2016-08-25T15:18:19-04:00 DBG scan path /var/log/mylogfile
2016-08-25T15:18:24-04:00 DBG Flushing spooler because of timeout. Events flushed: 0
When I should actually be seeing something like this:
> 2016-08-25T15:51:56-04:00 DBG Start next scan
> 2016-08-25T15:51:56-04:00 DBG scan path /var/log/mylogfile
> 2016-08-25T15:51:56-04:00 DBG Check file for harvesting: /var/log/mylogfile
> 2016-08-25T15:51:56-04:00 DBG Same file as before found. Fetch the state.
> 2016-08-25T15:51:56-04:00 DBG Update existing file for harvesting: /var/log/mylogfile
> 2016-08-25T15:51:56-04:00 DBG Not harvesting, file didn't change: /var/log/mylogfile
> 2016-08-25T15:51:59-04:00 DBG Flushing spooler because of timeout. Events flushed: 0
> 2016-08-25T15:52:01-04:00 DBG End of file reached: /var/log/mylogfile; Backoff now.
> 2016-08-25T15:52:06-04:00 DBG Flushing spooler because of timeout. Events flushed: 0
And I am getting some filebeat messages in the messages log file:
Aug 25 15:25:04 myvm /usr/bin/filebeat[7685]: prospector.go:133: Set backoff duration to 1s
Aug 25 15:25:04 myvm /usr/bin/filebeat[7685]: prospector.go:133: Set max_backoff duration to 10s
Aug 25 15:25:04 myvm /usr/bin/filebeat[7685]: prospector.go:113: force_close_file is disabled
Aug 25 15:25:04 myvm /usr/bin/filebeat[7685]: prospector.go:143: Starting prospector of type: log
Aug 25 15:25:04 myvm /usr/bin/filebeat[7685]: crawler.go:78: All prospectors initialised with 3 states to persist
Aug 25 15:25:04 myvm /usr/bin/filebeat[7685]: registrar.go:87: Starting Registrar
Aug 25 15:25:04 myvm /usr/bin/filebeat[7685]: publish.go:88: Start sending events to output
Aug 25 15:25:04 myvm /usr/bin/filebeat[7685]: spooler.go:77: Starting spooler: spool_size: 2048; idle_timeout: 5s
The log messages you posted above are only from the startup of filebeat. There should be mor log messages following, especially if you enabled debug level. Can you post these?
Thank you for your response and yes there are more messages but didn't feel they were relevant to the question I am asking. Ultimately, I think I still have a permissions issue with filebeat being run as a non-root user; I simply took a repeating loop from the log and only included one iteration of the loop. When trying to get my non-root user to function, the filebeat user fails to check the file for harvesting among other things. This is curious because I set a facl for read access to the log file. Does it need more than read access? Do you have guidance on what additional files/directories a non-root user would need to function correctly?
What happens if you become the filebeat user that you added (sudo su filebeat)? Can you read the log files? Can you read the directories containing the log files?
Filebeat needs to be able to stat the log file. According to the docs for stat, execute (search) permissions are required on any directories that lead to the file.
If you are familiar with strace, you could use it to look for permission errors in Filebeat.
Not sure if you have altered your logs here for security but it seems it is checking for /var/log/mylogfile.
Typically /var/log is owned by root and a non-root user won't be able to create files under /var/log. In my experience filebeat will rotate the logs and create a new filebeat log file with every restart. So you need to make sure that the filebeat user has write permissions to the log directory. Creating a directory /var/log/filebeat and having that directory be owned by filebeat should do the trick if this is the issue.
I run my filebeat with non-root user under RHEL with no issues. I am not using RPM deployment though. The way I accomplish it is by making sure the service start script runs as the non-root user so the start command
will get triggered by the non-root user and it all works fine.
Here is the checkuser function of my service start shell script:
RUN_AS="filebeat"
# if this script is not yet running as the right user use su to run it again as the correct user
# This will prevent output files from being created with wrong user ownership
checkuser() {
# Check to see if we are running as the right user. Else switch, and re-exec
if test `/usr/bin/id -un` = $RUN_AS; then
#echo $0 --- Already running as correct user $RUN_AS
:
elif test `/usr/bin/id -un` = "root"; then
echo "$0 --- Running as `/usr/bin/id -un`. Re-exec to run as $RUN_AS"
exec /bin/su - $RUN_AS -c "$0 $ARGS"
exit 0
else
echo "Only root or $RUN_AS can start $EXECUTABLE"
exit 1
fi
}
...
ARGS=$*;
case $1 in
'start')
checkuser
for CONFIG in "${config_list[@]}"; do
startproc
done
;;
...
@andrewkroh; yes I do have read permissions on the file and I did test that I was not aware of execute permissions being needed for collecting log files but adding that to the facl seemed to fix this for me. The only problem I seem to have now is filebeat is not giving verbose logging (see above) when set to debug. Any idea on what the root cause of this might be?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
I was not aware of execute permissions being needed for collecting log files but adding that to the facl seemed to fix this for me. The only problem I seem to have now is filebeat is not giving verbose logging (see above) when set to debug. Any idea on what the root cause of this might be?