Auditbeat randomly shuts down

jamesspi · June 11, 2018, 10:25am

Hi All,

I have an auditbeat instance running on RHEL 6, just the file integrity module. At certain times, it just stops running with the following error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xe606a5]

goroutine 82 [running]:
github.com/elastic/beats/vendor/github.com/joeshaw/multierror.(*MultiError).Error(0xc4200f6e00, 0x4, 0xc42a36980b)
	/go/src/github.com/elastic/beats/vendor/github.com/joeshaw/multierror/multierror.go:43 +0x95
github.com/elastic/beats/vendor/github.com/pkg/errors.(*withMessage).Error(0xc4200f6e20, 0xad9977, 0x1a8200f6f60)
	/go/src/github.com/elastic/beats/vendor/github.com/pkg/errors/errors.go:228 +0x37
github.com/elastic/beats/vendor/github.com/pkg/errors.(*withStack).Error(0xc4200f6e60, 0x20, 0x60000000020)
	<autogenerated>:1 +0x3c
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.encodeError(0x170bf0a, 0x5, 0x1f91640, 0xc4200f6e60, 0x1faffc0, 0xc4200f6f60, 0xc4200f6f40, 0xc4201bac00)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/error.go:46 +0x49
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.Field.AddTo(0x170bf0a, 0x5, 0x19, 0x0, 0x0, 0x0, 0x15f6fa0, 0xc4200f6e60, 0x1faffc0, 0xc4200f6f60)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/field.go:165 +0xc90
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.addFields(0x1faffc0, 0xc4200f6f60, 0xc420414500, 0x1, 0x2)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/field.go:199 +0x62
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.consoleEncoder.writeContext(0xc4204437c0, 0xc420601840, 0xc420414500, 0x1, 0x2)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/console_encoder.go:131 +0xcf
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.consoleEncoder.EncodeEntry(0xc4204437c0, 0x1, 0xbebef6c1af74e1ec, 0x4827658c562e, 0x1fe2ce0, 0x1714a5f, 0xe, 0x171b577, 0x16, 0x1, ...)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/console_encoder.go:110 +0x3de
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.(*ioCore).Write(0xc4203145d0, 0x1, 0xbebef6c1af74e1ec, 0x4827658c562e, 0x1fe2ce0, 0x1714a5f, 0xe, 0x171b577, 0x16, 0x1, ...)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/core.go:86 +0xa9
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc42007c580, 0xc420414500, 0x1, 0x2)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/entry.go:215 +0xe7
github.com/elastic/beats/vendor/go.uber.org/zap.(*SugaredLogger).log(0xc42000e190, 0xc420425001, 0x171b577, 0x16, 0x0, 0x0, 0x0, 0xc42004bdc8, 0x2, 0x2)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/sugar.go:234 +0xf6
github.com/elastic/beats/vendor/go.uber.org/zap.(*SugaredLogger).Warnw(0xc42000e190, 0x171b577, 0x16, 0xc42004bdc8, 0x2, 0x2)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/sugar.go:185 +0x83
github.com/elastic/beats/libbeat/logp.(*Logger).Warnw(0xc42000e1a0, 0x171b577, 0x16, 0xc42004bdc8, 0x2, 0x2)
	/go/src/github.com/elastic/beats/libbeat/logp/logger.go:121 +0x60
github.com/elastic/beats/auditbeat/module/file_integrity.(*reader).consumeEvents(0xc420010210, 0xc4202a69c0)
	/go/src/github.com/elastic/beats/auditbeat/module/file_integrity/eventreader_fsnotify.go:85 +0x62d
created by github.com/elastic/beats/auditbeat/module/file_integrity.(*reader).Start
	/go/src/github.com/elastic/beats/auditbeat/module/file_integrity/eventreader_fsnotify.go:54 +0x3dd

I have several other instances of RHEL 6, Centos 6, Centos 7, Windows 10 and Windows Server 2008R2 with no issue.

Any ideas?

Thanks!
James

adrisr · June 11, 2018, 1:53pm

Hi,

Which version of auditbeat is it?

jamesspi · June 11, 2018, 2:23pm

Hi @adrisr,

6.2.4.

Thanks!

adrisr · June 11, 2018, 3:08pm

Can you share your auditbeat configuration?

jamesspi · June 11, 2018, 3:10pm

Sure, here is a slightly redacted version:

    auditbeat.modules:
- module: file_integrity
  paths:
  - /home
  - /bin
  - /usr/bin
  - /sbin
  - /usr/sbin
  - /etc
  exclude_files:
  - '(?i)\.sw[nop]$'
  - '~$'
  - '/\.git($|/)'
  scan_at_start: true
  scan_rate_per_sec: 50 MiB
  max_file_size: 5000 MiB
  hash_types: [sha256]
  recursive: true
setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 1
  index.codec: best_compression
output.elasticsearch:
  index: "XXXXXXXXXXXXXXXXXXX"
  proxy_url: XXXXXXXXXXXXXXXXXX
cloud.id: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
cloud.auth: XXXXXXXXXXX:XXXXXXXXXXX
xpack.monitoring.enabled: true
setup.template.name: "auditbeat"
setup.template.pattern: "auditbeat-*"
setup.dashboards.enabled: false
setup.dashboards.index: "auditbeat-*"

Thanks,
James

adrisr · June 12, 2018, 7:14am

Can you run auditbeat with debug mode (-d '*') and attach the lines above the crash?

jamesspi · June 12, 2018, 7:15am

Sure, this is currently running in production so will need to raise a CR. Will keep you posted on progress.

jamesspi · June 12, 2018, 1:49pm

Hi @adrisr,

Here you go (slightly redacted again):

2018-06-12T10:00:11.462+0200	DEBUG	[file_integrity]	file_integrity/metricset.go:207	File changed since it was last seen	{"file_path": "/home/xxxxxx/xxxxxx/sedDk6pUG", "took": 10582, "event": {"old": null, "new": {"timestamp":"2018-06-12T08:00:11.457630097Z","path":"/home/xxxxxx/xxxxxx/sedDk6pUG","info":null,"source":"fsnotify","action":"attributes_modified"}}}
2018-06-12T10:00:11.463+0200	DEBUG	[file_integrity]	file_integrity/metricset.go:207	File changed since it was last seen	{"file_path": "/home/xxxx/xxxxx/sedDk6pUG", "took": 10913, "event": {"old": null, "new": {"timestamp":"2018-06-12T08:00:11.459856775Z","path":"/home/xxxxx/xxxxx/sedDk6pUG","info":null,"source":"fsnotify","action":"moved"}}}
2018-06-12T10:00:11.465+0200	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-06-12T08:00:11.457Z",
  "@metadata": {
    "beat": "auditbeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "beat": {
    "name": "xxxxx",
    "hostname": "xxxxx",
    "version": "6.2.4"
  },
  "event": {
    "module": "file_integrity",
    "action": [
      "attributes_modified"
    ]
  },
  "file": {
    "path": "/home/xxxxxx/xxxxx/sedDk6pUG"
  }
}
2018-06-12T10:00:11.465+0200	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-06-12T08:00:11.459Z",
  "@metadata": {
    "beat": "auditbeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "beat": {
    "name": "xxxxxx",
    "hostname": "xxxxxx",
    "version": "6.2.4"
  },
  "event": {
    "action": [
      "moved"
    ],
    "module": "file_integrity"
  },
  "file": {
    "path": "/home/xxxxx/xxxxx/sedDk6pUG"
  }
}
2018-06-12T10:00:11.470+0200	DEBUG	[module]	module/wrapper.go:177	Stopped metricSetWrapper[module=file_integrity, name=file, host=]
2018-06-12T10:00:11.470+0200	DEBUG	[module]	module/wrapper.go:120	Stopped Wrapper[name=file_integrity, len(metricSetWrappers)=1]
panic: runtime error: invalid memory address or nil pointer dereference

Thanks!
James

adrisr · June 13, 2018, 2:21am

While I still couldn't reproduce the issue, after some investigation I've found what could be the culprit.

Can you try this package to see if the problem is gone?

https://drive.google.com/file/d/1TArHDI3-GbZMm0BolxOBxorUACMJTLy_/view?usp=sharing

jamesspi · June 13, 2018, 6:54am

Thanks @adrisr! I'll give this a go and keep you posted.

James

adrisr · June 19, 2018, 4:37am

Hi @jamesspi

Did you have the chance to test it?

If it fixed the problem we would like to release it asap

jamesspi · June 19, 2018, 7:08am

Hi @adrisr,

Sorry, was away for a few days - raising a change to get this installed ASAP.

Thanks for following up!
James

adrisr · June 27, 2018, 8:34am

@jamesspi did it make a difference or still crashing?

jamesspi · June 28, 2018, 3:29pm

Hi Adrian,

Apologies for the late reply.

It didn’t happen again, so it looks like the fix worked

Thanks a mill!

James

adrisr · July 2, 2018, 7:01pm

@jamesspi can you tell me the exact version (and kernel version) for this RHEL6 box ? I'm still trying to reproduce it

jamesspi · July 4, 2018, 5:31pm

Hi Adrian,

Sorry for the slow responses, currently on vacation.

I’m getting the version for you through my ex colleague, as I am no longer with the same company. Will keep you posted.

Thanks!

James

adrisr · July 4, 2018, 6:26pm

Hi @jamesspi,

You can give my contact details to your ex-coworker so I don't have to bother you anymore

adrian.serrano at elastic.co

Salty · May 1, 2019, 5:10pm

Hi,
Just ran into this same issue.
Host: CentOS 6.10
Kernel: 2.6.32-754.12.1.el6.x86_64

I've tried Auditbeat v6.7.1 and v7.0.1 and they both randomly shutdown.

I tried the version you posted (v6.2.4_SNAPSHOT-1) and it hasn't crashed yet.

Is there anything I can do to help identify/test/verify the crash so the issue can be resolved with the newer clients?

Thanks

adrisr · May 2, 2019, 10:20am

The fix for this issue (#7753) is already merged into 6.7 / 7.0. You must be experiencing a different bug that is not present in 6.2.

Can you see any stack trace in the logs like the one in this thread?

Also share your auditbeat.yml contents.

Salty · May 2, 2019, 3:34pm

Hi,
I created a new CentOS 6.10 VM this morning and installed Auditbeat v6.7.1.

I upgraded the kernel but everything else is as CentOS shipped it.

Auditbeat.yml - pretty much a stock file.

This is an example configuration file highlighting only the most common

options. The auditbeat.reference.yml file from the same directory contains all

the supported options with more comments. You can use it as a reference.

You can find the full configuration reference here:

https://www.elastic.co/guide/en/beats/auditbeat/index.html

#========================== Modules configuration =============================
auditbeat.modules:

module: auditd

Load audit rules from separate files. Same format as audit.rules(7).

audit_rule_files: [ '${path.config}/audit.rules.d/*.conf' ]
audit_rules: |

Define audit rules here.

Create file watches (-w) or syscall audits (-a or -A). Uncomment these

examples or add your own rules.

If you are on a 64 bit platform, everything should be running

in 64 bit mode. This rule will detect any use of the 32 bit syscalls

because this might be a sign of someone exploiting a hole in the 32

bit API.

#-a always,exit -F arch=b32 -S all -F key=32bit-abi

Executions.

#-a always,exit -F arch=b64 -S execve,execveat -k exec

External access (warning: these can be expensive to audit).

#-a always,exit -F arch=b64 -S accept,bind,connect -F key=external-access

Identity changes.

-w /etc/group -p wa -k identity
-w /etc/passwd -p wa -k identity
-w /etc/gshadow -p wa -k identity

Unauthorized access attempts.

#-a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EACCES -k access
#-a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EPERM -k access
module: file_integrity
paths:
- /bin
- /usr/bin
- /sbin
- /usr/sbin
- /etc
module: system
datasets:
- host # General host information, e.g. uptime, IPs
- login # User logins, logouts, and system boots.
- package # Installed, updated, and removed packages
- process # Started and stopped processes
- socket # Opened and closed sockets
- user # User information
How often datasets send state updates with the

current state of the system (e.g. all currently

running processes, all open sockets).

state.period: 12h

Enabled by default. Auditbeat will read password fields in

/etc/passwd and /etc/shadow and store a hash locally to

detect any changes.

user.detect_password_changes: true

File patterns of the login record files.

login.wtmp_file_pattern: /var/log/wtmp*
login.btmp_file_pattern: /var/log/btmp*

#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 3
#index.codec: best_compression
#_source.enabled: false

#================================ General =====================================

The name of the shipper that publishes the network data. It can be used to group

all the transactions sent by a single shipper in the web interface.

#name:

The tags of the shipper are included in their own field with each

transaction published.

#tags: ["service-X", "web-tier"]

Optional fields that you can specify to add additional information to the

output.

#fields:

env: staging

#============================== Dashboards =====================================

These settings control loading the sample dashboards to the Kibana index. Loading

the dashboards is disabled by default and can be enabled either by setting the

options here, or by using the `-setup` CLI flag or the `setup` command.

#setup.dashboards.enabled: false

The URL from where to download the dashboards archive. By default this URL

has a value which is computed based on the Beat name and version. For released

versions, this URL points to the dashboard archive on the artifacts.elastic.co

website.

#setup.dashboards.url:

#============================== Kibana =====================================

Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.

This requires a Kibana endpoint configuration.

setup.kibana:

Kibana Host

Scheme and port can be left out and will be set to the default (http and 5601)

In case you specify and additional path, the scheme is required: http://localhost:5601/path

IPv6 addresses should always be defined as: https://[2001:db8::1]:5601

#host: "localhost:5601"

Kibana Space ID

ID of the Kibana Space into which the dashboards should be loaded. By default,

the Default Space will be used.

#space.id:

#============================= Elastic Cloud ==================================

These settings simplify using auditbeat with the Elastic Cloud (https://cloud.elastic.co/).

The cloud.id setting overwrites the `output.elasticsearch.hosts` and

`setup.kibana.host` options.

You can find the `cloud.id` in the Elastic Cloud web UI.

#cloud.id:

The cloud.auth setting overwrites the `output.elasticsearch.username` and

`output.elasticsearch.password` settings. The format is `<user>:<pass>`.

#cloud.auth:

#================================ Outputs =====================================

Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:

Array of hosts to connect to.

hosts: ["192.168.153.177:9200"]

Enabled ilm (beta) to use index lifecycle management instead daily indices.

#ilm.enabled: false

Optional protocol and basic auth credentials.

#protocol: "https"
#username: "elastic"
#password: "changeme"

#----------------------------- Logstash output --------------------------------
#output.logstash:

The Logstash hosts

#hosts: ["localhost:5044"]

Optional SSL. By default is off.

List of root certificates for HTTPS server verifications

#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

Certificate for SSL client authentication

#ssl.certificate: "/etc/pki/client/cert.pem"

Client Certificate Key

#ssl.key: "/etc/pki/client/cert.key"

#================================ Processors =====================================

Configure processors to enhance or manipulate events generated by the beat.

processors:

add_host_metadata: ~
add_cloud_metadata: ~

#================================ Logging =====================================

Sets log level. The default log level is info.

Available log levels are: error, warning, info, debug

#logging.level: debug

At debug level, you can selectively enable logging only for some components.

To enable all selectors use ["*"]. Examples of other selectors are "beat",

"publish", "service".

#logging.selectors: ["*"]

Topic		Replies	Views
Crashing 5.0.0-alpha1 Beats filebeat	2	832	July 5, 2017
[Filebeat : v7.9.3] Invalid memory address or nil pointer dereference Beats filebeat	4	317	October 6, 2022
Filebeat crash each day Beats filebeat	2	494	August 28, 2019
Error : invalid memory address or nil pointer dereference Beats filebeat	6	2363	July 26, 2016
64Bit alignment issue on ARM7 (Raspberry Pi) Beats filebeat	7	2205	January 30, 2017

Auditbeat randomly shuts down

This is an example configuration file highlighting only the most common

options. The auditbeat.reference.yml file from the same directory contains all

the supported options with more comments. You can use it as a reference.

You can find the full configuration reference here:

https://www.elastic.co/guide/en/beats/auditbeat/index.html

Load audit rules from separate files. Same format as audit.rules(7).

Define audit rules here.

Create file watches (-w) or syscall audits (-a or -A). Uncomment these

examples or add your own rules.

If you are on a 64 bit platform, everything should be running

in 64 bit mode. This rule will detect any use of the 32 bit syscalls

because this might be a sign of someone exploiting a hole in the 32

bit API.

Executions.

External access (warning: these can be expensive to audit).

Identity changes.

Unauthorized access attempts.

How often datasets send state updates with the

current state of the system (e.g. all currently

running processes, all open sockets).

Enabled by default. Auditbeat will read password fields in

/etc/passwd and /etc/shadow and store a hash locally to

detect any changes.

File patterns of the login record files.

The name of the shipper that publishes the network data. It can be used to group

all the transactions sent by a single shipper in the web interface.

The tags of the shipper are included in their own field with each

transaction published.

Optional fields that you can specify to add additional information to the

output.

env: staging

These settings control loading the sample dashboards to the Kibana index. Loading

the dashboards is disabled by default and can be enabled either by setting the

options here, or by using the -setup CLI flag or the setup command.

The URL from where to download the dashboards archive. By default this URL

has a value which is computed based on the Beat name and version. For released

versions, this URL points to the dashboard archive on the artifacts.elastic.co

website.

Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.

This requires a Kibana endpoint configuration.

Kibana Host

Scheme and port can be left out and will be set to the default (http and 5601)

In case you specify and additional path, the scheme is required: http://localhost:5601/path

IPv6 addresses should always be defined as: https://[2001:db8::1]:5601

Kibana Space ID

ID of the Kibana Space into which the dashboards should be loaded. By default,

the Default Space will be used.

These settings simplify using auditbeat with the Elastic Cloud (https://cloud.elastic.co/).

The cloud.id setting overwrites the output.elasticsearch.hosts and

setup.kibana.host options.

You can find the cloud.id in the Elastic Cloud web UI.

The cloud.auth setting overwrites the output.elasticsearch.username and

output.elasticsearch.password settings. The format is <user>:<pass>.

Configure what output to use when sending the data collected by the beat.

Array of hosts to connect to.

Enabled ilm (beta) to use index lifecycle management instead daily indices.

Optional protocol and basic auth credentials.

The Logstash hosts

Optional SSL. By default is off.

List of root certificates for HTTPS server verifications

Certificate for SSL client authentication

Client Certificate Key

Configure processors to enhance or manipulate events generated by the beat.

Sets log level. The default log level is info.

Available log levels are: error, warning, info, debug

At debug level, you can selectively enable logging only for some components.

To enable all selectors use ["*"]. Examples of other selectors are "beat",

"publish", "service".

Related topics

options here, or by using the `-setup` CLI flag or the `setup` command.

The cloud.id setting overwrites the `output.elasticsearch.hosts` and

`setup.kibana.host` options.

You can find the `cloud.id` in the Elastic Cloud web UI.

The cloud.auth setting overwrites the `output.elasticsearch.username` and

`output.elasticsearch.password` settings. The format is `<user>:<pass>`.