# Dec 21st, 2024: \[EN\] Ingesting syslog and auth logs from Journald into Elastic Stack with Elastic-Agent or Beats
**URL:** https://discuss.elastic.co/t/dec-21st-2024-en-ingesting-syslog-and-auth-logs-from-journald-into-elastic-stack-with-elastic-agent-or-beats/371047
**Category:** Advent Calendar
**Created:** [December 21, 2024, 8:00am UTC](https://discuss.elastic.co/t/dec-21st-2024-en-ingesting-syslog-and-auth-logs-from-journald-into-elastic-stack-with-elastic-agent-or-beats/371047 "2024-12-21T08:00:00Z")
**Posts on this page:** 1
**Page:** 1
### Author:  [@TiagoQueiroz](https://discuss.elastic.co/u/TiagoQueiroz)
#### Post date: [December 21, 2024, 8:00am UTC](https://discuss.elastic.co/t/dec-21st-2024-en-ingesting-syslog-and-auth-logs-from-journald-into-elastic-stack-with-elastic-agent-or-beats/371047/1 "2024-12-21T08:00:00Z")
We have been working hard at Elastic to add more Linux distributions to our supported OS matrix, and support for Debian 12 has just been added for Elastic-Agent and Beats!
This post is a demo of the features we're developing to add support for Linux distributions that use [Journald](https://0pointer.de/blog/projects/journalctl.html) to store the system and auth logs.
Some Linux distributions like Debian 12, have completely moved away from traditional log files for system logs, so the only way to get them now is by reading the journal.
# Ingesting the system logs with Elastic-Agent
The [system integration](https://www.elastic.co/docs/current/en/integrations/system) for Elastic-Agent collects system logs and metrics from your servers, starting with version `1.63.0` (and Elastic-Agent `8.17.0`) it will also support ingesting the logs from Journald.
We will use a Debian 12 VM as example here. However if you want to collect system logs with Journald from other distributions, it will also work after a little configuration tweeking.
**TL;DR:** Just install Elastic-Agent `8.17.0` on a Debian 12 VM add the system integration, and you'll have the logs coming from Journald. If you install the Elastic-Agent on another distribution, like Debian 11, it will read the log files. It is that simple.
**The longer version**
There is no magic, just some clever utilisation of our features, before jumping into the "how to" part, let's first understand how this works.
The `syslog` and `auth` data streams from the system integration now have a new input, journald that, in its default configuration, will only run on Debian 12 and Amazon Linux 2023. The current log input will do the opposite and not run on Debian 12 and Amazon Linux 2023.
The Elastic-Agent integrations support the use of [conditions](https://www.elastic.co/guide/en/fleet/current/dynamic-input-configuration.html#conditions) to decide whether to run an input or not, this is already used by the system integration to prevent the `winlog` input to run on Linux or MacOS. Now we're using it to decide in which Linux distributions to run the journald or log inputs.
If you look at the integration configuration you will see a new `conditions` field pre-populated, for the Journald input it looks like this:
```auto
${host.os_version} == "12 (bookworm)" or (${host.os_platform} == "amzn" and ${host.os_version} == "2023")
```
If this condition evaluates to `true`, then the input will run, if not, the input won't run.
To run the input in a different Linux distribution just edit the condition to match your criteria. Leaving it empty will always run the input.
_Be careful! If you run both, the log and journald input on Linux distributions that use both traditional log files and journald, you will duplicate data and the system dashboards will also show duplicated values!_
Here is how the integration configuration looks like:
When you're done configuring the integration (or accepting the defaults), select whether you want to add it to a new policy or to an existing one:
Then click on "Save and continue" and follow to "Add Elastic Agent to your hosts"
Then follow the instructions to add the agent, using the Linux Tar, you will need to run the following commands on your host:
```auto
curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.17.0-linux-x86_64.tar.gz
tar xzvf elastic-agent-8.17.0-linux-x86_64.tar.gz
cd elastic-agent-8.17.0-linux-x86_64
sudo ./elastic-agent install --url=https://:8220 --enrollment-token=
```
Once the Elastic-Agent is installed, you can go the its overview page and you'll see that the journald input is running and the log input is not available:
## How can I tailor the `condition` for my hosts?
That's easy, the [host provider](https://www.elastic.co/guide/en/fleet/current/host-provider.html) documentation list they keys provided by it, as you saw in the example we're using `host.os_version` and `host.os_platform`, if you want to check the exact values reported by your host, in Kibana go to Management -\> Fleet -\> Agents -\> -\> Diagnostics then click on "Request diagnostics .zip", once the diagnostics are ready, download and extract the zip archive. Look for `variables.yml`, under the key `host` you'll have all keys and values reported by the host provider, e.g:
```yaml
host:
architecture: x86_64
id: ad88a1859979427ea1a7c24f0ae0320a
ip:
- 127.0.0.1/8
- ::1/128
mac:
- 08:00:27:5e:8a:a5
name: debian12
os_family: debian
os_platform: debian
os_version: 12 (bookworm)
platform: linux
```
# What about standalone Beats?
I'm glad you asked! Yes, Filebeat also supports Debian 12 or any Linux distribution that uses Journald, the [system module](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-system.html) supports Journald starting in `8.17.0`.
For Filebeat things are a little as configurations are per-host and you need to edit the configuration files manually. Given that, all you need to do is to enable the system module and edit the configuration from both filesets (`syslog` and `auth`) to set `var.use_journald: true`
1. Make sure you follow the steps as root as you will need root access to read parts of the journal.
2. Download and extract the `tar.gz` package
3. Configure `filebeat.yml` with your Elasticsearch and Kibana credentials (you need the Kibana credentials to setup the data views, dashboards, etc.
4. Ensure modules are enabled:
```auto
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
reload.period: 10s
```
5. Test the connection to the output by running `./filebeat test output`:
```auto
root@Debian12:~/filebeat-8.17.0-linux-x86_64# ./filebeat test output
elasticsearch: https://advent-calendar-deployment.elastic-cloud.com:443...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: 35.235.72.223
dial up... OK
TLS...
security: server's certificate chain verification is enabled
handshake... OK
TLS version: TLSv1.3
dial up... OK
talk to server... OK
version: 8.17.0
root@Debian12:~/filebeat-8.17.0-linux-x86_64#
```
6. Enable the system module `./filebeat modules enable system`:
```auto
root@Debian12:~/filebeat-8.17.0-linux-x86_64# ./filebeat modules enable system
Enabled system
root@Debian12:~/filebeat-8.17.0-linux-x86_64#
```
7. Edit `./modules.d/system.yml to enable the filesets and journald:
```yaml
# Module: system
# Docs: https://www.elastic.co/guide/en/beats/filebeat/8.x/filebeat-module-system.html
- module: system
syslog:
enabled: true
var.use_journald: true
auth:
enabled: true
var.use_journald: true
```
8. Setup the system module `./filebeat setup --modules system`
9. Run Filebeat `./filebeat -e -v`. The `-e -v` flags will make log at level info to `stderr`.
10. Go to "Discover" in Kibana and select the `filebeat-*` data view, you should see events containing `event.dataset: system.syslog` and `event.dataset: system.auth`.
11. Head to "Dashboards" and search for `"[Logs System]"` (use the quotes to filter out the Windows dashboard). All four dashboards will be working, just make sure to set the appropriated time slice, the default 15min is not enough on some hosts.