Auditbeat randomly shuts down

Hi All,

I have an auditbeat instance running on RHEL 6, just the file integrity module. At certain times, it just stops running with the following error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xe606a5]

goroutine 82 [running]:*MultiError).Error(0xc4200f6e00, 0x4, 0xc42a36980b)
	/go/src/ +0x95*withMessage).Error(0xc4200f6e20, 0xad9977, 0x1a8200f6f60)
	/go/src/ +0x37*withStack).Error(0xc4200f6e60, 0x20, 0x60000000020)
	<autogenerated>:1 +0x3c, 0x5, 0x1f91640, 0xc4200f6e60, 0x1faffc0, 0xc4200f6f60, 0xc4200f6f40, 0xc4201bac00)
	/go/src/ +0x49, 0x5, 0x19, 0x0, 0x0, 0x0, 0x15f6fa0, 0xc4200f6e60, 0x1faffc0, 0xc4200f6f60)
	/go/src/ +0xc90, 0xc4200f6f60, 0xc420414500, 0x1, 0x2)
	/go/src/ +0x62, 0xc420601840, 0xc420414500, 0x1, 0x2)
	/go/src/ +0xcf, 0x1, 0xbebef6c1af74e1ec, 0x4827658c562e, 0x1fe2ce0, 0x1714a5f, 0xe, 0x171b577, 0x16, 0x1, ...)
	/go/src/ +0x3de*ioCore).Write(0xc4203145d0, 0x1, 0xbebef6c1af74e1ec, 0x4827658c562e, 0x1fe2ce0, 0x1714a5f, 0xe, 0x171b577, 0x16, 0x1, ...)
	/go/src/ +0xa9*CheckedEntry).Write(0xc42007c580, 0xc420414500, 0x1, 0x2)
	/go/src/ +0xe7*SugaredLogger).log(0xc42000e190, 0xc420425001, 0x171b577, 0x16, 0x0, 0x0, 0x0, 0xc42004bdc8, 0x2, 0x2)
	/go/src/ +0xf6*SugaredLogger).Warnw(0xc42000e190, 0x171b577, 0x16, 0xc42004bdc8, 0x2, 0x2)
	/go/src/ +0x83*Logger).Warnw(0xc42000e1a0, 0x171b577, 0x16, 0xc42004bdc8, 0x2, 0x2)
	/go/src/ +0x60*reader).consumeEvents(0xc420010210, 0xc4202a69c0)
	/go/src/ +0x62d
created by*reader).Start
	/go/src/ +0x3dd

I have several other instances of RHEL 6, Centos 6, Centos 7, Windows 10 and Windows Server 2008R2 with no issue.

Any ideas?



Which version of auditbeat is it?

Hi @adrisr,



Can you share your auditbeat configuration?

Sure, here is a slightly redacted version:

- module: file_integrity
  - /home
  - /bin
  - /usr/bin
  - /sbin
  - /usr/sbin
  - /etc
  - '(?i)\.sw[nop]$'
  - '~$'
  - '/\.git($|/)'
  scan_at_start: true
  scan_rate_per_sec: 50 MiB
  max_file_size: 5000 MiB
  hash_types: [sha256]
  recursive: true
  index.number_of_shards: 1
  index.number_of_replicas: 1
  index.codec: best_compression
xpack.monitoring.enabled: true "auditbeat"
setup.template.pattern: "auditbeat-*"
setup.dashboards.enabled: false
setup.dashboards.index: "auditbeat-*"


Can you run auditbeat with debug mode (-d '*') and attach the lines above the crash?

Sure, this is currently running in production so will need to raise a CR. Will keep you posted on progress.

Hi @adrisr,

Here you go (slightly redacted again):

2018-06-12T10:00:11.462+0200	DEBUG	[file_integrity]	file_integrity/metricset.go:207	File changed since it was last seen	{"file_path": "/home/xxxxxx/xxxxxx/sedDk6pUG", "took": 10582, "event": {"old": null, "new": {"timestamp":"2018-06-12T08:00:11.457630097Z","path":"/home/xxxxxx/xxxxxx/sedDk6pUG","info":null,"source":"fsnotify","action":"attributes_modified"}}}
2018-06-12T10:00:11.463+0200	DEBUG	[file_integrity]	file_integrity/metricset.go:207	File changed since it was last seen	{"file_path": "/home/xxxx/xxxxx/sedDk6pUG", "took": 10913, "event": {"old": null, "new": {"timestamp":"2018-06-12T08:00:11.459856775Z","path":"/home/xxxxx/xxxxx/sedDk6pUG","info":null,"source":"fsnotify","action":"moved"}}}
2018-06-12T10:00:11.465+0200	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-06-12T08:00:11.457Z",
  "@metadata": {
    "beat": "auditbeat",
    "type": "doc",
    "version": "6.2.4"
  "beat": {
    "name": "xxxxx",
    "hostname": "xxxxx",
    "version": "6.2.4"
  "event": {
    "module": "file_integrity",
    "action": [
  "file": {
    "path": "/home/xxxxxx/xxxxx/sedDk6pUG"
2018-06-12T10:00:11.465+0200	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-06-12T08:00:11.459Z",
  "@metadata": {
    "beat": "auditbeat",
    "type": "doc",
    "version": "6.2.4"
  "beat": {
    "name": "xxxxxx",
    "hostname": "xxxxxx",
    "version": "6.2.4"
  "event": {
    "action": [
    "module": "file_integrity"
  "file": {
    "path": "/home/xxxxx/xxxxx/sedDk6pUG"
2018-06-12T10:00:11.470+0200	DEBUG	[module]	module/wrapper.go:177	Stopped metricSetWrapper[module=file_integrity, name=file, host=]
2018-06-12T10:00:11.470+0200	DEBUG	[module]	module/wrapper.go:120	Stopped Wrapper[name=file_integrity, len(metricSetWrappers)=1]
panic: runtime error: invalid memory address or nil pointer dereference


While I still couldn't reproduce the issue, after some investigation I've found what could be the culprit.

Can you try this package to see if the problem is gone?

Thanks @adrisr! I'll give this a go and keep you posted.


Hi @jamesspi

Did you have the chance to test it?

If it fixed the problem we would like to release it asap

Hi @adrisr,

Sorry, was away for a few days - raising a change to get this installed ASAP.

Thanks for following up!

@jamesspi did it make a difference or still crashing?

Hi Adrian,

Apologies for the late reply.

It didn’t happen again, so it looks like the fix worked :slight_smile:

Thanks a mill!


@jamesspi can you tell me the exact version (and kernel version) for this RHEL6 box ? I'm still trying to reproduce it

Hi Adrian,

Sorry for the slow responses, currently on vacation.

I’m getting the version for you through my ex colleague, as I am no longer with the same company. Will keep you posted.



Hi @jamesspi,

You can give my contact details to your ex-coworker so I don't have to bother you anymore :slight_smile:

adrian.serrano at

Just ran into this same issue.
Host: CentOS 6.10
Kernel: 2.6.32-754.12.1.el6.x86_64

I've tried Auditbeat v6.7.1 and v7.0.1 and they both randomly shutdown.

I tried the version you posted (v6.2.4_SNAPSHOT-1) and it hasn't crashed yet. :+1:

Is there anything I can do to help identify/test/verify the crash so the issue can be resolved with the newer clients?


The fix for this issue (#7753) is already merged into 6.7 / 7.0. You must be experiencing a different bug that is not present in 6.2.

Can you see any stack trace in the logs like the one in this thread?

Also share your auditbeat.yml contents.

I created a new CentOS 6.10 VM this morning and installed Auditbeat v6.7.1.

I upgraded the kernel but everything else is as CentOS shipped it.

Auditbeat.yml - pretty much a stock file.

This is an example configuration file highlighting only the most common

options. The auditbeat.reference.yml file from the same directory contains all

the supported options with more comments. You can use it as a reference.

You can find the full configuration reference here:

#========================== Modules configuration =============================

  • module: auditd

    Load audit rules from separate files. Same format as audit.rules(7).

    audit_rule_files: [ '${path.config}/audit.rules.d/*.conf' ]
    audit_rules: |

    Define audit rules here.

    Create file watches (-w) or syscall audits (-a or -A). Uncomment these

    examples or add your own rules.

    If you are on a 64 bit platform, everything should be running

    in 64 bit mode. This rule will detect any use of the 32 bit syscalls

    because this might be a sign of someone exploiting a hole in the 32

    bit API.

    #-a always,exit -F arch=b32 -S all -F key=32bit-abi


    #-a always,exit -F arch=b64 -S execve,execveat -k exec

    External access (warning: these can be expensive to audit).

    #-a always,exit -F arch=b64 -S accept,bind,connect -F key=external-access

    Identity changes.

    -w /etc/group -p wa -k identity
    -w /etc/passwd -p wa -k identity
    -w /etc/gshadow -p wa -k identity

    Unauthorized access attempts.

    #-a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EACCES -k access
    #-a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EPERM -k access

  • module: file_integrity

    • /bin
    • /usr/bin
    • /sbin
    • /usr/sbin
    • /etc
  • module: system

    • host # General host information, e.g. uptime, IPs
    • login # User logins, logouts, and system boots.
    • package # Installed, updated, and removed packages
    • process # Started and stopped processes
    • socket # Opened and closed sockets
    • user # User information

    How often datasets send state updates with the

    current state of the system (e.g. all currently

    running processes, all open sockets).

    state.period: 12h

    Enabled by default. Auditbeat will read password fields in

    /etc/passwd and /etc/shadow and store a hash locally to

    detect any changes.

    user.detect_password_changes: true

    File patterns of the login record files.

    login.wtmp_file_pattern: /var/log/wtmp*
    login.btmp_file_pattern: /var/log/btmp*

#==================== Elasticsearch template setting ==========================
index.number_of_shards: 3
#index.codec: best_compression
#_source.enabled: false

#================================ General =====================================

The name of the shipper that publishes the network data. It can be used to group

all the transactions sent by a single shipper in the web interface.


The tags of the shipper are included in their own field with each

transaction published.

#tags: ["service-X", "web-tier"]

Optional fields that you can specify to add additional information to the



env: staging

#============================== Dashboards =====================================

These settings control loading the sample dashboards to the Kibana index. Loading

the dashboards is disabled by default and can be enabled either by setting the

options here, or by using the -setup CLI flag or the setup command.

#setup.dashboards.enabled: false

The URL from where to download the dashboards archive. By default this URL

has a value which is computed based on the Beat name and version. For released

versions, this URL points to the dashboard archive on the



#============================== Kibana =====================================

Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.

This requires a Kibana endpoint configuration.


Kibana Host

Scheme and port can be left out and will be set to the default (http and 5601)

In case you specify and additional path, the scheme is required: http://localhost:5601/path

IPv6 addresses should always be defined as: https://[2001:db8::1]:5601

#host: "localhost:5601"

Kibana Space ID

ID of the Kibana Space into which the dashboards should be loaded. By default,

the Default Space will be used.

#============================= Elastic Cloud ==================================

These settings simplify using auditbeat with the Elastic Cloud (

The setting overwrites the output.elasticsearch.hosts and options.

You can find the in the Elastic Cloud web UI.

The cloud.auth setting overwrites the output.elasticsearch.username and

output.elasticsearch.password settings. The format is <user>:<pass>.


#================================ Outputs =====================================

Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------

Array of hosts to connect to.

hosts: [""]

Enabled ilm (beta) to use index lifecycle management instead daily indices.

#ilm.enabled: false

Optional protocol and basic auth credentials.

#protocol: "https"
#username: "elastic"
#password: "changeme"

#----------------------------- Logstash output --------------------------------

The Logstash hosts

#hosts: ["localhost:5044"]

Optional SSL. By default is off.

List of root certificates for HTTPS server verifications

#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

Certificate for SSL client authentication

#ssl.certificate: "/etc/pki/client/cert.pem"

Client Certificate Key

#ssl.key: "/etc/pki/client/cert.key"

#================================ Processors =====================================

Configure processors to enhance or manipulate events generated by the beat.


  • add_host_metadata: ~
  • add_cloud_metadata: ~

#================================ Logging =====================================

Sets log level. The default log level is info.

Available log levels are: error, warning, info, debug

#logging.level: debug

At debug level, you can selectively enable logging only for some components.

To enable all selectors use ["*"]. Examples of other selectors are "beat",

"publish", "service".

#logging.selectors: ["*"]