Aggregating and sending multiple error logs to FileBeat 6


(Aaron) #1

Hi,

I have a WHM server with multiple accounts. On each account I have error logs from PHP. I would like to send all error logs to FileBeat for processing and reading.

There is a big issue though... I don't know where all the error logs are on the server since they are in sub-directories.

Currently, the only way I can find all error logs is by running the following command - find /home -type f -name error_log -exec du -sh {} \;. This searches the home directory for any error log files and outputs them to the screen with their file sizes. So I thought of two different ways to handle the data with Filebeat.

  1. Aggregate all error log files into one log file every day
  2. Create symlinks to the files every day to a general directory that Filebeat will see. For example -
    find /home -type f -name error_log -exec bash -c 'ln -s "$1" "$(mktemp -u XXXXXX)"' _ {} \; (Directory might be /var/log/apache2/user_errors)

Although, aggregation won't work because the errors might be inputted into ES(Elastic search) multiple times because the file is being created every day based upon the cron log. The second might work, but I couldn't get symlinks working properly with FileBeat. I kept on running into the error that it couldn't read the files even though I enabled symlinks in my configuration. Plus I don't think it is a good solution since I will have to run the cron job everyday to clean up the symlinks and check for any new files each day thus possibly inputting in the same data into the ES.

Any other thoughts on this matter?

Thanks a lot!


(Pier-Hugues Pellerin) #2

Hello,

I would probably use some wildcards to define the path where filebeat is expecting to find the logs.

I assume there is some constant in the path where the error logs are created, maybe something like the following would work for your case?

/home/*/*/logs/error_logs

Thanks


(Aaron) #3

That is the issue really. There is no consistency with error_logs. For example, here are a few of the log file directories -

  • /home/john/public_html/error_log
  • /home/john/public_html/wp-content/themes/alyeska/error_log
  • /home/john/public_html/wp-content/themes/twentyfifteen/error_log
  • /home/john/public_html/wp-content/themes/twentysixteen/error_log
  • /home/john/public_html/wp-content/themes/twentyfourteen/error_log
  • /home/john/test.john.com/login/error_log

Thanks!
Aaron


(Pier-Hugues Pellerin) #4

Oh I feel the pain :slight_smile:

Do you have a lot of users on that machine?


(Aaron) #5

@pierhugues - Yes, sadly. That is why I was hoping to aggregate them all into one or symlink the error logs to a main directory. cringe


(Pier-Hugues Pellerin) #6

I was thinking about using filebeat's recursive glob, but I am afraid of performances when scanning for new logs.

Symlinks should works, what error did you run into?


(Aaron) #7

Well, first I was planning on running this command find /home -type f -name error_log -exec bash -c 'ln -s "$1" "$(mktemp -u XXXXXX)"' _ {} \; in a cron job every day to add or remove error logs. I didn't know if filebeat would then try to re-index the config files if the symlink name changed. I was hoping it wouldn't re-index the whole file, and just index the data that is new.

The next error I keep on running into is - File /var/log/apache2/user_errors/wgn8cs skipped as it is a symlink.


(Aaron) #8

Here are my config files -
filebeats.yml -

Summary
filebeat.prospectors:

- type: log
  
  enabled: false
  
  paths:
    - /var/log/*.log
    

  symlinks: true

filebeat.config.modules:
  
  path: ${path.config}/modules.d/*.yml
  
  reload.enabled: false
  
  

setup.template.settings:
  index.number_of_shards: 3


name: Name


filebeat.registry_file: ${path.data}/registry




setup.dashboards.enabled: true

setup.kibana:

  host: "http://IP:5601"


output.elasticsearch:
  
  hosts: ["IP:9200"]

logging.level: debug



logging.selectors: ["*"]

My apache2.yml -

Summary
- module: apache2
  # Access logs
  access:
    enabled: false

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/apache2/access_log"]

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/apache2/error_log", "/var/log/apache2/user_errors/*"]

(Pier-Hugues Pellerin) #9

Symlinks, It should be fine because Filebeat uses the inode+offset of the file, each new file will get a new inode and will make Filebeat reread it.

I see the symlink problem in your configuration; you are enabling the symlinks on your custom prospector log, you have to configure it in the apache2 module.

Something like this:

- module: apache2
  # Access logs
  access:
    enabled: false

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/apache2/access_log"]

  # Error logs
  error:
    enabled: true
    prospector.symlinks: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/apache2/error_log", "/var/log/apache2/user_errors/*"]

Modules are just sugar on top of the log harvester, but you can override any prospectors settings, see this doc.

Thanks


(Aaron) #10

@pierhugues - You sir, are a genius. Thanks for the help! I have been playing around with that all weekend long trying to get it to work properly with symlinks. That should do it. I owe you a beer :slight_smile:


(system) #11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.