Elastic Agent 9.1.3 – system_audit.package on Rocky Linux 9.6 – no DNF install/remove events

Elastic Stack Elastic Agent
1

Hi,

I’m trying to collect package installation/removal events using the system_audit.package stream on Rocky Linux 9.6.

I’m running Elastic Agent 9.1.3 (elastic-agent-complete) in a Podman container.
image: ``docker.elastic.co/elastic-agent/elastic-agent-complete:9.1.3

I am using Kibana System Audit Integration v1.11.0.

A relevant part of my elastic-agent.yml input configuration looks like this:

  - id: system_audit-audit/system
    type: audit/system
    streams:
      - id: audit/system-system_audit.package
        type: audit/system
        data_stream:
          dataset: system_audit.package
          type: logs
        datasets:
          - package
        period: 2m
        state.period: 12h
        tags:
          - audit-system-package
        processors:
          - add_host_metadata: null

The Elastic Agent container is running via Podman with the following mounts:

- /etc/machine-id:/hostfs/etc/machine-id:ro
- /var/lib/rpm:/hostfs/var/lib/rpm:ro
- /var/lib/dnf:/hostfs/var/lib/dnf:ro
- /var/log/dnf.log:/hostfs/var/log/dnf.log:ro,z
- /var/log/dnf.rpm.log:/hostfs/var/log/dnf.rpm.log:ro,z

In Kibana, I can see the following:

CleanShot 2025-11-07 at 12.40.25
CleanShot 2025-11-07 at 12.40.252543×1198 239 KB

Test sequence (waited >15 min; period: 2m):

dnf install -y telnet vim
dnf remove -y telnet vim
dnf install -y telnet vim

The Elastic Agent logs show no errors — everything looks normal, but no package install/remove events are being collected.

I verified all DNF/RPM mounts inside the Elastic Agent container:

podman exec -it elastic-agent bash -lc '
echo "=== /etc/machine-id ===" &&
cat /hostfs/etc/machine-id 2>/dev/null || echo "missing or unreadable"

echo -e "\n=== /var/lib/rpm ===" &&
ls -lh /hostfs/var/lib/rpm || echo "missing"

echo -e "\n=== /var/lib/dnf ===" &&
ls -lh /hostfs/var/lib/dnf || echo "missing"

echo -e "\n=== /var/log/dnf.log ===" &&
tail -n 5 /hostfs/var/log/dnf.log 2>/dev/null || echo "missing"

echo -e "\n=== /var/log/dnf.rpm.log ===" &&
tail -n 5 /hostfs/var/log/dnf.rpm.log 2>/dev/null || echo "missing"

There is output:

=== /etc/machine-id ===
e8ae9f280dd64481a75b6ae279c36f8c

=== /var/lib/rpm ===
total 86M
-rw-r--r--. 1 root root 86M Nov  7 11:21 rpmdb.sqlite
-rw-r--r--. 1 root root 32K Nov  7 11:44 rpmdb.sqlite-shm
-rw-r--r--. 1 root root   0 Nov  7 11:21 rpmdb.sqlite-wal

=== /var/lib/dnf ===
total 4.4M
-rw-r--r--. 1 root root 380K Nov  7 11:21 history.sqlite
-rw-r--r--. 1 root root  32K Nov  7 11:21 history.sqlite-shm
-rw-r--r--. 1 root root 4.0M Nov  7 11:21 history.sqlite-wal
drwxr-xr-x. 7 root root 4.0K Sep  1 11:25 repos

=== /var/log/dnf.log ===
2025-11-07T12:44:33+0100 DEBUG cachedir: /var/cache/dnf
2025-11-07T12:44:33+0100 CRITICAL No such command: %{_dbpath}. Please use /usr/bin/dnf --help
2025-11-07T12:44:33+0100 CRITICAL It could be a DNF plugin command, try: "dnf install 'dnf-command(%{_dbpath})'"
2025-11-07T12:44:33+0100 DDEBUG Cleaning up.
2025-11-07T12:44:33+0100 DDEBUG Plugins were unloaded.

=== /var/log/dnf.rpm.log ===
2025-11-07T12:21:02+0100 INFO --- logging initialized ---
2025-11-07T12:21:07+0100 INFO --- logging initialized ---
2025-11-07T12:21:22+0100 SUBDEBUG Installed: telnet-1:0.17-85.el9.x86_64
2025-11-07T12:41:06+0100 INFO --- logging initialized ---
2025-11-07T12:44:33+0100 INFO --- logging initialized ---

Question:
I don’t see any DNF install/remove events — does the system_audit.package stream actually support collecting them on Rocky Linux 9.6, or does it only report the current package state?

2

Hi,

Yes, install/remove events do work. I just tested this setup - Rocky Linux 9.6, agent in podman. One important point is that for this to work, the agent must see the changes in /var/lib/rpm so I mapped:
-v /etc/machine-id:/etc/machine-id:ro
-v /var/lib/rpm:/var/lib/rpm:ro

Another point is that if you delete and install right after, this rapid A → B → A change can be missed in our polling. Decrease the polling rate from default 2m in the integration to lower and it will likely catch more such scenarios. Test with single install or single remove and you should see the corresponding events in the dashboard.

1 Like
3

Thanks a lot Michale :+1: I installed and removed the package within just a few seconds — that’s probably why I don’t see any change.

4

I tested it by installing the cowsay package and removing telnet, then waited about 10 minutes, but no changes appeared. I’m not sure why — I tried it using both YUM and DNF.

Do you have any tips on how to resolve this?

5

@vasek Did you change your mounts as suggested here? Rather than /hostfs/….

1 Like
6

Ah, I see — I thought /hostfs was recommended on Linux to keep the host filesystem logically separated from the container’s, so I used:
environment:
SYSTEM_HOSTFS: /hostfs
ELASTIC_AGENT_HOSTFS: /hostfs

But it seems Elastic Agent doesn’t actually use these variables in this case.

Thanks again for the clarification — the issue really was the mount itself. I just wanted to keep the setup cleaner without modifying the container filesystem too much.

CleanShot 2025-11-11 at 08.56.55
CleanShot 2025-11-11 at 08.56.551591×864 80.7 KB

7

Glad to have helped.

Out of curiosity, why don’t you just monitor the host (packages) directly? Doing it via an agent in a container seems to me a bit, er, over-engineered?

Maybe there’s something I’m missing here … ?

8

You’re right, it does add another layer :slightly_smiling_face:

I just wanted to keep things clearly separated — the Linux team takes care of the OS, and the application team manages Elastic.

I ran into a few cases where it worked on some machines but not on others because I had changed something in the OS here and there. Running it in a container just keeps everything cleaner and more consistent. Plus, it’s portable and reproducible — I can spin it up anywhere, and it works the same every time.

9

Ok. Thanks for clarification.

I understand your point, I just disagree with it. My view is that Its just moving the work around, doing stuff “because we can” rather than because of real substantive benefit. Often driven by what are effectively turf wars. Better to just work collaboratively, in my view. The more complex a solution becomes, the harder it is to maintain. This thread might be thought to demonstrates my point.

But, I don’t walk in your shoes.

Thanks again.

10

I get your point, and I agree that simplicity is important.

For me, the main benefit is portability and consistency. Responsibilities often split between teams, and environments tend to drift over time.

By containerizing the agent and managing everything through Ansible, I can ensure it’s consistent, reproducible, and isolated — no surprises, no leftover configs, and the same setup everywhere :slight_smile: