I have a WHM server with multiple accounts. On each account I have error logs from PHP. I would like to send all error logs to FileBeat for processing and reading.
There is a big issue though... I don't know where all the error logs are on the server since they are in sub-directories.
Currently, the only way I can find all error logs is by running the following command - find /home -type f -name error_log -exec du -sh {} \;. This searches the home directory for any error log files and outputs them to the screen with their file sizes. So I thought of two different ways to handle the data with Filebeat.
Aggregate all error log files into one log file every day
Create symlinks to the files every day to a general directory that Filebeat will see. For example - find /home -type f -name error_log -exec bash -c 'ln -s "$1" "$(mktemp -u XXXXXX)"' _ {} \; (Directory might be /var/log/apache2/user_errors)
Although, aggregation won't work because the errors might be inputted into ES(Elastic search) multiple times because the file is being created every day based upon the cron log. The second might work, but I couldn't get symlinks working properly with FileBeat. I kept on running into the error that it couldn't read the files even though I enabled symlinks in my configuration. Plus I don't think it is a good solution since I will have to run the cron job everyday to clean up the symlinks and check for any new files each day thus possibly inputting in the same data into the ES.
Well, first I was planning on running this command find /home -type f -name error_log -exec bash -c 'ln -s "$1" "$(mktemp -u XXXXXX)"' _ {} \; in a cron job every day to add or remove error logs. I didn't know if filebeat would then try to re-index the config files if the symlink name changed. I was hoping it wouldn't re-index the whole file, and just index the data that is new.
The next error I keep on running into is - File /var/log/apache2/user_errors/wgn8cs skipped as it is a symlink.
- module: apache2
# Access logs
access:
enabled: false
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/var/log/apache2/access_log"]
# Error logs
error:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/var/log/apache2/error_log", "/var/log/apache2/user_errors/*"]
Symlinks, It should be fine because Filebeat uses the inode+offset of the file, each new file will get a new inode and will make Filebeat reread it.
I see the symlink problem in your configuration; you are enabling the symlinks on your custom prospector log, you have to configure it in the apache2 module.
Something like this:
- module: apache2
# Access logs
access:
enabled: false
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/var/log/apache2/access_log"]
# Error logs
error:
enabled: true
prospector.symlinks: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/var/log/apache2/error_log", "/var/log/apache2/user_errors/*"]
Modules are just sugar on top of the log harvester, but you can override any prospectors settings, see this doc.
@pierhugues - You sir, are a genius. Thanks for the help! I have been playing around with that all weekend long trying to get it to work properly with symlinks. That should do it. I owe you a beer
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.