Unable to start auditbeat on docker

Elastic Stack Beats
1

Hi, I've been trying to deploy the official auditbeat:6.2.4 container on our centos 7.4 but I'm stuck with the following error log:

2018-05-21T03:53:08.825Z INFO instance/beat.go:468 Home path: [/usr/share/auditbeat] Config path: [/usr/share/auditbeat] Data path: [/usr/share/auditbeat/data] Logs path: [/usr/share/auditbeat/logs]
2018-05-21T03:53:08.825Z DEBUG [beat] instance/beat.go:495 Beat metadata path: /usr/share/auditbeat/data/meta.json
2018-05-21T03:53:08.825Z INFO instance/beat.go:475 Beat UUID: 0bd41a5d-0893-4d51-9ffc-1a4d1eb95fa6
2018-05-21T03:53:08.825Z INFO instance/beat.go:213 Setup Beat: auditbeat; Version: 6.2.4
2018-05-21T03:53:08.825Z DEBUG [beat] instance/beat.go:230 Initializing output plugins
2018-05-21T03:53:08.825Z DEBUG [processors] processors/processor.go:49 Processors:
2018-05-21T03:53:08.825Z INFO elasticsearch/client.go:145 Elasticsearch url: http://10.4.95.242:9200
2018-05-21T03:53:08.826Z INFO pipeline/module.go:76 Beat name: 84e65b21e37f
2018-05-21T03:53:08.826Z DEBUG [modules] beater/metricbeat.go:80 Register [ModuleFactory:, MetricSetFactory:[auditd/auditd, file_integrity/file]]
2018-05-21T03:53:08.826Z DEBUG [processors] processors/processor.go:49 Processors:
2018-05-21T03:53:08.826Z INFO [auditd] auditd/audit_linux.go:65 auditd module is running as euid=0 on kernel=3.10.0-693.el7.x86_64
2018-05-21T03:53:08.826Z ERROR instance/beat.go:667 Exiting: 1 error: 1 error: failed to create audit client: failed to get audit status: failed sending request: connection refused
Exiting: 1 error: 1 error: failed to create audit client: failed to get audit status: failed sending request: connection refused

I have disabled the auditd,

● auditd.service - Security Auditing Service
Loaded: loaded (/usr/lib/systemd/system/auditd.service; disabled; vendor preset: enabled)
Active: inactive (dead) since Mon 2018-05-21 13:05:56 AEST; 58min ago
Docs: man:auditd(8)
GitHub - linux-audit/audit-documentation: Documentation and specifications
Main PID: 707 (code=exited, status=0/SUCCESS)

I've also used the following params to start the container as suggested,

docker run -d --name sandbox_auditbeat --privileged --pid=host -v /tmp/auditbeat.yml:/usr/share/auditbeat/auditbeat.yml docker.elastic.co/beats/auditbeat:6.2.4

As seen from the output above, the kernel is 3.10 which means the unicast socket type is used so it should work just fine as long as we have disabled the auditd; doesn't seem to be the case here.

The auditbeat.yml,

auditbeat.modules:
- module: auditd
  audit_rules: |
    -a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EACCES -k access
    -a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EPERM -k access
fields:
  tenant: abc
setup.kibana:
  host: "10.4.95.244:5601"
output.elasticsearch:
  hosts: ["10.4.95.242:9200"]
  username: "elastic"
  password: "changeme"
logging.level: debug

What else could I've missed here?

2

If you also have auditd installed on this box, can you trying running sudo auditctl -s and sudo auditctl -l on this box and report the output?

What OS is this? Is it a custom kernel build?

There are couple reasons for "connection refused" (aka ECONNREFUSED) that I could find from various sources.

  • The ECONNREFUSED error can indicate that the kernel has the audit system disabled.
  • It can indicate that the process is running in a different user namespace.
3

Here's the output

[root@xxx ~]# auditctl -s
enabled 1
failure 1
pid 0
rate_limit 0
backlog_limit 8192
lost 228
backlog 0
loginuid_immutable 0 unlocked
[root@xxx ~]# auditctl -l
No rules

It's a standard CentOS 7.4,

[root@xxx ~]# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
[root@xxx ~]# uname -a
Linux xxx.domain 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

I can see the auditd output just fine when it is enabled so it shouldn't be the kernel I reckon. I've installed the auditbeat directly onto the OS instead of using the official container and it runs just fine apparently. So it must've been something to do with the docker/image setting?

As for the namespace, I've already put the "--pid=host" option and set privilege mode to it as instructed on the manual which should have addressed the user namespace issue.

4

It looks like you have done all the right things AFAICT. I don't see anything obvious in the kernel sources for v3.10.0 (though I didn't look at the patches applied by RH) that would cause ECONNREFUSED. What about looking in the logs. Is there anything related in dmesg from the kernel?

Not sure if this command works in the journald version on Centos 7, but is there anything interesting in sudo journalctl -af _TRANSPORT=audit?

5

The commands work but there's really nothing that relates. Nothing comes up when I started the container.

Just to reiterate, when I installed the auditbeat directly on the host (instead of container), it works just fine. There's no error messages whatsoever.

Only when I use the official container then it fails,

6

I'm running into this problem as well.

Host OS: Centos - Linux 3.10
Docker container: Centos - Linux 3.10
Auditbeat 6.2.4
I'm running with --cap-add=ALL --pid=host and --privileged

Auditbeat runs fine on the host. When I try to run Auditbeat in a container I get this error:

Exiting: 1 error: 1 error: failed to create audit client: failed to get audit status: failed sending request: connection refused

7

It looks like there's no official Auditbeat image that we can use. Is that correct?

I didn't see an Auditbeat image here:

Is there a known reason why it may not work in Docker?

8

It looks like it's failing in audit_linux.go right here :

// process be in initial PID namespace).
status, err := ms.client.GetStatus()

I run it with --cap-add=AUDIT_CONTROL and --pid=host so it seems it should work.

9

I added --net=host and then got a slightly different error:

operation not permitted

It looks like there's some permission error. I'll investigate further.

10

It works if I run as root and --net=host.

2 Likes