Moving to the Beat: Filebeat config updates in 6.x
Hi everyone 
! I am relatively new to the Elastic. Consequently, a good portion of my time has been devoted to shoring up my knowledge, filling the gaps between self-taught and formally trained. I found this information along the way about one of my favorite Elastic Utilities: Filebeat.
By design, Filebeat is simple to configure and lightweight. Want to ship some logs? No problem, here's how you set it up:
filebeat.prospectors:
- type: log
paths:
- /var/log/nginx/access.log*
exclude_files: ['\.gz$']
- type: log
paths:
- /var/log/messages
- /var/log/*.log
It's not especially glamorous, it gets the job done and moves log data. If you want to get rich data, it needs to cycle through Logstash to be interpreted. But, I found out I can make the setup process easier!
I ran into the Filebeat modules when I was re-familiarizing myself with 5.X (ironically just weeks before the 6.0 release) and loved how simple it was to set up shipping of some common logs to Elasticsearch and load basic dashboards to get started on a new project quickly.
First, set up your dashboards (this uses your output setting for Elasticsearch in filbeat.yml, you'll receive an error if this is incorrect or missing):
$ ./filebeat -e -modules=nginx -setup
That automatically configures sample dashboards for the nginx module, so it only needs to be run once for the module dashboards you want to load. Then you can configure the modules through the config file in the future:
filebeat.modules:
- module: nginx
You can distribute that config using some configuration management (like Chef, Ansible, or Puppet) and get sane outputs for Nginx out of the box! It's very handy for quickly configuring some data output or managing minimally complex configurations using an Elasticsearch ingest node. As you can imagine, though, that single filebeat.yml file could get quite large. And you might need a separate config for multiple server types.
In 6.x, though, configuration got a little more manageable! Now, the Filebeat modules are all stored in separate configuration files located in filebeat/modules.d/
$ ls modules.d
apache2.yml.disabled icinga.yml.disabled
nginx.yml.disabled system.yml.disabled
auditd.yml.disabled mysql.yml.disabled
redis.yml.disabled
From the output above, you can see the modules start by default as *.disabled, but they already contain some default settings. Simply renaming the file to remove disabled will load the module with default settings which alone allows you to quickly test and iterate on your server configuration settings.
Furthermore, "setup" can be run as a standalone command instead of a flag when starting Filebeat, making it easier to handle from an operator's machine with the following command:
filebeat setup --modules MODULE_LIST --dashboards
This builds your Kibana Dashboards only — there’s no ingesting unwanted data, or having to separate your setup steps from your server configs.
All of this doubles to make distribution of your configuration across the environment way easier. Using some configuration management, your servers can now have a common filebeat.yml for configuring common settings (like output and TLS), and include your filebeat/modules.d/*.yml based on roles. You can configure your Elastic Stack dashboards from an operator computer and then write your configurations once, instead of rewriting a full filebeat.yml for each system.
All granular details for the current module configurations can be found in our modules documentation, but I always learn by getting hands-on with some new tech. So whether you prefer reading the technical specifications or messing with a new test config, I hope I’ve inspired you to continuously improve your Elastic Stack configurations this holiday season!