Auditbeat randomly shuts down

Elastic Stack Beats
1

Hi All,

I have an auditbeat instance running on RHEL 6, just the file integrity module. At certain times, it just stops running with the following error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xe606a5]

goroutine 82 [running]:
github.com/elastic/beats/vendor/github.com/joeshaw/multierror.(*MultiError).Error(0xc4200f6e00, 0x4, 0xc42a36980b)
	/go/src/github.com/elastic/beats/vendor/github.com/joeshaw/multierror/multierror.go:43 +0x95
github.com/elastic/beats/vendor/github.com/pkg/errors.(*withMessage).Error(0xc4200f6e20, 0xad9977, 0x1a8200f6f60)
	/go/src/github.com/elastic/beats/vendor/github.com/pkg/errors/errors.go:228 +0x37
github.com/elastic/beats/vendor/github.com/pkg/errors.(*withStack).Error(0xc4200f6e60, 0x20, 0x60000000020)
	<autogenerated>:1 +0x3c
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.encodeError(0x170bf0a, 0x5, 0x1f91640, 0xc4200f6e60, 0x1faffc0, 0xc4200f6f60, 0xc4200f6f40, 0xc4201bac00)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/error.go:46 +0x49
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.Field.AddTo(0x170bf0a, 0x5, 0x19, 0x0, 0x0, 0x0, 0x15f6fa0, 0xc4200f6e60, 0x1faffc0, 0xc4200f6f60)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/field.go:165 +0xc90
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.addFields(0x1faffc0, 0xc4200f6f60, 0xc420414500, 0x1, 0x2)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/field.go:199 +0x62
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.consoleEncoder.writeContext(0xc4204437c0, 0xc420601840, 0xc420414500, 0x1, 0x2)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/console_encoder.go:131 +0xcf
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.consoleEncoder.EncodeEntry(0xc4204437c0, 0x1, 0xbebef6c1af74e1ec, 0x4827658c562e, 0x1fe2ce0, 0x1714a5f, 0xe, 0x171b577, 0x16, 0x1, ...)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/console_encoder.go:110 +0x3de
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.(*ioCore).Write(0xc4203145d0, 0x1, 0xbebef6c1af74e1ec, 0x4827658c562e, 0x1fe2ce0, 0x1714a5f, 0xe, 0x171b577, 0x16, 0x1, ...)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/core.go:86 +0xa9
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc42007c580, 0xc420414500, 0x1, 0x2)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/entry.go:215 +0xe7
github.com/elastic/beats/vendor/go.uber.org/zap.(*SugaredLogger).log(0xc42000e190, 0xc420425001, 0x171b577, 0x16, 0x0, 0x0, 0x0, 0xc42004bdc8, 0x2, 0x2)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/sugar.go:234 +0xf6
github.com/elastic/beats/vendor/go.uber.org/zap.(*SugaredLogger).Warnw(0xc42000e190, 0x171b577, 0x16, 0xc42004bdc8, 0x2, 0x2)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/sugar.go:185 +0x83
github.com/elastic/beats/libbeat/logp.(*Logger).Warnw(0xc42000e1a0, 0x171b577, 0x16, 0xc42004bdc8, 0x2, 0x2)
	/go/src/github.com/elastic/beats/libbeat/logp/logger.go:121 +0x60
github.com/elastic/beats/auditbeat/module/file_integrity.(*reader).consumeEvents(0xc420010210, 0xc4202a69c0)
	/go/src/github.com/elastic/beats/auditbeat/module/file_integrity/eventreader_fsnotify.go:85 +0x62d
created by github.com/elastic/beats/auditbeat/module/file_integrity.(*reader).Start
	/go/src/github.com/elastic/beats/auditbeat/module/file_integrity/eventreader_fsnotify.go:54 +0x3dd

I have several other instances of RHEL 6, Centos 6, Centos 7, Windows 10 and Windows Server 2008R2 with no issue.

Any ideas?

Thanks!
James

2

Hi,

Which version of auditbeat is it?

3

Hi @adrisr,

6.2.4.

Thanks!

4

Can you share your auditbeat configuration?

5

Sure, here is a slightly redacted version:

    auditbeat.modules:
- module: file_integrity
  paths:
  - /home
  - /bin
  - /usr/bin
  - /sbin
  - /usr/sbin
  - /etc
  exclude_files:
  - '(?i)\.sw[nop]$'
  - '~$'
  - '/\.git($|/)'
  scan_at_start: true
  scan_rate_per_sec: 50 MiB
  max_file_size: 5000 MiB
  hash_types: [sha256]
  recursive: true
setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 1
  index.codec: best_compression
output.elasticsearch:
  index: "XXXXXXXXXXXXXXXXXXX"
  proxy_url: XXXXXXXXXXXXXXXXXX
cloud.id: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
cloud.auth: XXXXXXXXXXX:XXXXXXXXXXX
xpack.monitoring.enabled: true
setup.template.name: "auditbeat"
setup.template.pattern: "auditbeat-*"
setup.dashboards.enabled: false
setup.dashboards.index: "auditbeat-*"

Thanks,
James

6

Can you run auditbeat with debug mode (-d '*') and attach the lines above the crash?

7

Sure, this is currently running in production so will need to raise a CR. Will keep you posted on progress.

8

Hi @adrisr,

Here you go (slightly redacted again):

2018-06-12T10:00:11.462+0200	DEBUG	[file_integrity]	file_integrity/metricset.go:207	File changed since it was last seen	{"file_path": "/home/xxxxxx/xxxxxx/sedDk6pUG", "took": 10582, "event": {"old": null, "new": {"timestamp":"2018-06-12T08:00:11.457630097Z","path":"/home/xxxxxx/xxxxxx/sedDk6pUG","info":null,"source":"fsnotify","action":"attributes_modified"}}}
2018-06-12T10:00:11.463+0200	DEBUG	[file_integrity]	file_integrity/metricset.go:207	File changed since it was last seen	{"file_path": "/home/xxxx/xxxxx/sedDk6pUG", "took": 10913, "event": {"old": null, "new": {"timestamp":"2018-06-12T08:00:11.459856775Z","path":"/home/xxxxx/xxxxx/sedDk6pUG","info":null,"source":"fsnotify","action":"moved"}}}
2018-06-12T10:00:11.465+0200	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-06-12T08:00:11.457Z",
  "@metadata": {
    "beat": "auditbeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "beat": {
    "name": "xxxxx",
    "hostname": "xxxxx",
    "version": "6.2.4"
  },
  "event": {
    "module": "file_integrity",
    "action": [
      "attributes_modified"
    ]
  },
  "file": {
    "path": "/home/xxxxxx/xxxxx/sedDk6pUG"
  }
}
2018-06-12T10:00:11.465+0200	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-06-12T08:00:11.459Z",
  "@metadata": {
    "beat": "auditbeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "beat": {
    "name": "xxxxxx",
    "hostname": "xxxxxx",
    "version": "6.2.4"
  },
  "event": {
    "action": [
      "moved"
    ],
    "module": "file_integrity"
  },
  "file": {
    "path": "/home/xxxxx/xxxxx/sedDk6pUG"
  }
}
2018-06-12T10:00:11.470+0200	DEBUG	[module]	module/wrapper.go:177	Stopped metricSetWrapper[module=file_integrity, name=file, host=]
2018-06-12T10:00:11.470+0200	DEBUG	[module]	module/wrapper.go:120	Stopped Wrapper[name=file_integrity, len(metricSetWrappers)=1]
panic: runtime error: invalid memory address or nil pointer dereference

Thanks!
James

9

While I still couldn't reproduce the issue, after some investigation I've found what could be the culprit.

Can you try this package to see if the problem is gone?

https://drive.google.com/file/d/1TArHDI3-GbZMm0BolxOBxorUACMJTLy_/view?usp=sharing

10

Thanks @adrisr! I'll give this a go and keep you posted.

James

11

Hi @jamesspi

Did you have the chance to test it?

If it fixed the problem we would like to release it asap

12

Hi @adrisr,

Sorry, was away for a few days - raising a change to get this installed ASAP.

Thanks for following up!
James

13

@jamesspi did it make a difference or still crashing?

14

Hi Adrian,

Apologies for the late reply.

It didn’t happen again, so it looks like the fix worked :slight_smile:

Thanks a mill!

James

15

@jamesspi can you tell me the exact version (and kernel version) for this RHEL6 box ? I'm still trying to reproduce it

16

Hi Adrian,

Sorry for the slow responses, currently on vacation.

I’m getting the version for you through my ex colleague, as I am no longer with the same company. Will keep you posted.

Thanks!

James

17

Hi @jamesspi,

You can give my contact details to your ex-coworker so I don't have to bother you anymore :slight_smile:

adrian.serrano at elastic.co

18

Hi,
Just ran into this same issue.
Host: CentOS 6.10
Kernel: 2.6.32-754.12.1.el6.x86_64

I've tried Auditbeat v6.7.1 and v7.0.1 and they both randomly shutdown.

I tried the version you posted (v6.2.4_SNAPSHOT-1) and it hasn't crashed yet. :+1:

Is there anything I can do to help identify/test/verify the crash so the issue can be resolved with the newer clients?

Thanks

19

The fix for this issue (#7753) is already merged into 6.7 / 7.0. You must be experiencing a different bug that is not present in 6.2.

Can you see any stack trace in the logs like the one in this thread?

Also share your auditbeat.yml contents.

20

Hi,
I created a new CentOS 6.10 VM this morning and installed Auditbeat v6.7.1.

I upgraded the kernel but everything else is as CentOS shipped it.

Auditbeat.yml - pretty much a stock file.

This is an example configuration file highlighting only the most common

options. The auditbeat.reference.yml file from the same directory contains all

the supported options with more comments. You can use it as a reference.

You can find the full configuration reference here:

https://www.elastic.co/guide/en/beats/auditbeat/index.html

#========================== Modules configuration =============================
auditbeat.modules:

  • module: auditd

    Load audit rules from separate files. Same format as audit.rules(7).

    audit_rule_files: [ '${path.config}/audit.rules.d/*.conf' ]
    audit_rules: |

    Define audit rules here.

    Create file watches (-w) or syscall audits (-a or -A). Uncomment these

    examples or add your own rules.

    If you are on a 64 bit platform, everything should be running

    in 64 bit mode. This rule will detect any use of the 32 bit syscalls

    because this might be a sign of someone exploiting a hole in the 32

    bit API.

    #-a always,exit -F arch=b32 -S all -F key=32bit-abi

    Executions.

    #-a always,exit -F arch=b64 -S execve,execveat -k exec

    External access (warning: these can be expensive to audit).

    #-a always,exit -F arch=b64 -S accept,bind,connect -F key=external-access

    Identity changes.

    -w /etc/group -p wa -k identity
    -w /etc/passwd -p wa -k identity
    -w /etc/gshadow -p wa -k identity

    Unauthorized access attempts.

    #-a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EACCES -k access
    #-a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EPERM -k access

  • module: file_integrity
    paths:

    • /bin
    • /usr/bin
    • /sbin
    • /usr/sbin
    • /etc

  • module: system
    datasets:

    • host # General host information, e.g. uptime, IPs
    • login # User logins, logouts, and system boots.
    • package # Installed, updated, and removed packages
    • process # Started and stopped processes
    • socket # Opened and closed sockets
    • user # User information

    How often datasets send state updates with the

    current state of the system (e.g. all currently

    running processes, all open sockets).

    state.period: 12h

    Enabled by default. Auditbeat will read password fields in

    /etc/passwd and /etc/shadow and store a hash locally to

    detect any changes.

    user.detect_password_changes: true

    File patterns of the login record files.

    login.wtmp_file_pattern: /var/log/wtmp*
    login.btmp_file_pattern: /var/log/btmp*

#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 3
#index.codec: best_compression
#_source.enabled: false

#================================ General =====================================

The name of the shipper that publishes the network data. It can be used to group

all the transactions sent by a single shipper in the web interface.

#name:

The tags of the shipper are included in their own field with each

transaction published.

#tags: ["service-X", "web-tier"]

Optional fields that you can specify to add additional information to the

output.

#fields:

env: staging

#============================== Dashboards =====================================

These settings control loading the sample dashboards to the Kibana index. Loading

the dashboards is disabled by default and can be enabled either by setting the

options here, or by using the -setup CLI flag or the setup command.

#setup.dashboards.enabled: false

The URL from where to download the dashboards archive. By default this URL

has a value which is computed based on the Beat name and version. For released

versions, this URL points to the dashboard archive on the artifacts.elastic.co

website.

#setup.dashboards.url:

#============================== Kibana =====================================

Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.

This requires a Kibana endpoint configuration.

setup.kibana:

Kibana Host

Scheme and port can be left out and will be set to the default (http and 5601)

In case you specify and additional path, the scheme is required: http://localhost:5601/path

IPv6 addresses should always be defined as: https://[2001:db8::1]:5601

#host: "localhost:5601"

Kibana Space ID

ID of the Kibana Space into which the dashboards should be loaded. By default,

the Default Space will be used.

#space.id:

#============================= Elastic Cloud ==================================

These settings simplify using auditbeat with the Elastic Cloud (https://cloud.elastic.co/).

The cloud.id setting overwrites the output.elasticsearch.hosts and

setup.kibana.host options.

You can find the cloud.id in the Elastic Cloud web UI.

#cloud.id:

The cloud.auth setting overwrites the output.elasticsearch.username and

output.elasticsearch.password settings. The format is <user>:<pass>.

#cloud.auth:

#================================ Outputs =====================================

Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:

Array of hosts to connect to.

hosts: ["192.168.153.177:9200"]

Enabled ilm (beta) to use index lifecycle management instead daily indices.

#ilm.enabled: false

Optional protocol and basic auth credentials.

#protocol: "https"
#username: "elastic"
#password: "changeme"

#----------------------------- Logstash output --------------------------------
#output.logstash:

The Logstash hosts

#hosts: ["localhost:5044"]

Optional SSL. By default is off.

List of root certificates for HTTPS server verifications

#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

Certificate for SSL client authentication

#ssl.certificate: "/etc/pki/client/cert.pem"

Client Certificate Key

#ssl.key: "/etc/pki/client/cert.key"

#================================ Processors =====================================

Configure processors to enhance or manipulate events generated by the beat.

processors:

  • add_host_metadata: ~
  • add_cloud_metadata: ~

#================================ Logging =====================================

Sets log level. The default log level is info.

Available log levels are: error, warning, info, debug

#logging.level: debug

At debug level, you can selectively enable logging only for some components.

To enable all selectors use ["*"]. Examples of other selectors are "beat",

"publish", "service".

#logging.selectors: ["*"]