We have been working hard at Elastic to add more Linux distributions to our supported OS matrix, and support for Debian 12 has just been added for Elastic-Agent and Beats!
This post is a demo of the features we're developing to add support for Linux distributions that use Journald to store the system and auth logs.
Some Linux distributions like Debian 12, have completely moved away from traditional log files for system logs, so the only way to get them now is by reading the journal.
Ingesting the system logs with Elastic-Agent
The system integration for Elastic-Agent collects system logs and metrics from your servers, starting with version 1.63.0 (and Elastic-Agent 8.17.0) it will also support ingesting the logs from Jounrald.
We will use a Debian 12 VM as example here. However if you want to collect system logs with Journald from other distributions, it will also work after a little configuration tweeking.
TL;DR: Just install Elastic-Agent 8.17.0 on a Debian 12 VM add the system integration, and you'll have the logs coming from Journald. If you install the Elastic-Agent on another distribution, like Debian 11, it will read the log files. It is that simple.
The longer version
There is no magic, just some clever utilisation of our features, before jumping into the "how to" part, let's first understand how this works.
The syslog and auth data streams from the system integration now have a new input, journald that, in its default configuration, will only run on Debian 12 and Amazon Linux 2023. The current log input will do the opposite and not run on Debian 12 and Amazon Linux 2023.
The Elastic-Agent integrations support the use of conditions to decide whether to run an input or not, this is already used by the system integration to prevent the winlog input to run on Linux or MacOS. Now we're using it to decide in which Linux distributions to run the journald or log inputs.
If you look at the integration configuration you will see a new conditions field pre-populated, for the Journald input it looks like this:
${host.os_version} == "12 (bookworm)" or (${host.os_platform} == "amzn" and ${host.os_version} == "2023")
If this condition evaluates to true, then the input will run, if not, the input won't run.
To run the input in a different Linux distribution just edit the condition to match your criteria. Leaving it empty will always run the input.
Be careful! If you run both, the log and journald input on Linux distributions that use both traditional log files and journald, you will duplicate data and the system dashboards will also show duplicated values!
Here is how the integration configuration looks like:
When you're done configuring the integration (or accepting the defaults), select whether you want to add it to a new policy or to an existing one:
Then click on "Save and continue" and follow to "Add Elastic Agent to your hosts"
Then follow the instructions to add the agent, using the Linux Tar, you will need to run the following commands on your host:
curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.17.0-linux-x86_64.tar.gz
tar xzvf elastic-agent-8.17.0-linux-x86_64.tar.gz
cd elastic-agent-8.17.0-linux-x86_64
sudo ./elastic-agent install --url=https://<YOUR FLEET SERVER ADDRESS>:8220 --enrollment-token=<YOUR ENROLMENT TOKEN>
Once the Elastic-Agent is installed, you can go the its overview page and you'll see that the journald input is running and the log input is not available:
How can I tailor the condition for my hosts?
That's easy, the host provider documentation list they keys provided by it, as you saw in the example we're using host.os_version and host.os_platform, if you want to check the exact values reported by your host, in Kibana go to Management -> Fleet -> Agents -> -> Diagnostics then click on "Request diagnostics .zip", once the diagnostics are ready, download and extract the zip archive. Look for variables.yml, under the key host you'll have all keys and values reported by the host provider, e.g:
host:
architecture: x86_64
id: ad88a1859979427ea1a7c24f0ae0320a
ip:
- 127.0.0.1/8
- ::1/128
mac:
- 08:00:27:5e:8a:a5
name: debian12
os_family: debian
os_platform: debian
os_version: 12 (bookworm)
platform: linux
What about standalone Beats?
I'm glad you asked! Yes, Filebeat also supports Debian 12 or any Linux distribution that uses Journald, the system module supports Journald starting in 8.17.0.
For Filebeat things are a little as configurations are per-host and you need to edit the configuration files manually. Given that, all you need to do is to enable the system module and edit the configuration from both filesets (syslog and auth) to set var.use_journald: true
- Make sure you follow the steps as root as you will need root access to read parts of the journal.
- Download and extract the
tar.gzpackage - Configure
filebeat.ymlwith your Elasticsearch and Kibana credentials (you need the Kibana credentials to setup the data views, dashboards, etc. - Ensure modules are enabled:
filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false reload.period: 10s - Test the connection to the output by running
./filebeat test output:root@Debian12:~/filebeat-8.17.0-linux-x86_64# ./filebeat test output elasticsearch: https://advent-calendar-deployment.elastic-cloud.com:443... parse url... OK connection... parse host... OK dns lookup... OK addresses: 35.235.72.223 dial up... OK TLS... security: server's certificate chain verification is enabled handshake... OK TLS version: TLSv1.3 dial up... OK talk to server... OK version: 8.17.0 root@Debian12:~/filebeat-8.17.0-linux-x86_64# - Enable the system module
./filebeat modules enable system:root@Debian12:~/filebeat-8.17.0-linux-x86_64# ./filebeat modules enable system Enabled system root@Debian12:~/filebeat-8.17.0-linux-x86_64# - Edit `./modules.d/system.yml to enable the filesets and journald:
# Module: system # Docs: https://www.elastic.co/guide/en/beats/filebeat/8.x/filebeat-module-system.html - module: system syslog: enabled: true var.use_journald: true auth: enabled: true var.use_journald: true - Setup the system module
./filebeat setup --modules system - Run Filebeat
./filebeat -e -v. The-e -vflags will make log at level info tostderr. - Go to "Discover" in Kibana and select the
filebeat-*data view, you should see events containingevent.dataset: system.syslogandevent.dataset: system.auth. - Head to "Dashboards" and search for
"[Logs System]"(use the quotes to filter out the Windows dashboard). All four dashboards will be working, just make sure to set the appropriated time slice, the default 15min is not enough on some hosts.