Auditbeat randomly shuts down


(Jamesspi) #1

Hi All,

I have an auditbeat instance running on RHEL 6, just the file integrity module. At certain times, it just stops running with the following error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xe606a5]

goroutine 82 [running]:
github.com/elastic/beats/vendor/github.com/joeshaw/multierror.(*MultiError).Error(0xc4200f6e00, 0x4, 0xc42a36980b)
	/go/src/github.com/elastic/beats/vendor/github.com/joeshaw/multierror/multierror.go:43 +0x95
github.com/elastic/beats/vendor/github.com/pkg/errors.(*withMessage).Error(0xc4200f6e20, 0xad9977, 0x1a8200f6f60)
	/go/src/github.com/elastic/beats/vendor/github.com/pkg/errors/errors.go:228 +0x37
github.com/elastic/beats/vendor/github.com/pkg/errors.(*withStack).Error(0xc4200f6e60, 0x20, 0x60000000020)
	<autogenerated>:1 +0x3c
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.encodeError(0x170bf0a, 0x5, 0x1f91640, 0xc4200f6e60, 0x1faffc0, 0xc4200f6f60, 0xc4200f6f40, 0xc4201bac00)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/error.go:46 +0x49
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.Field.AddTo(0x170bf0a, 0x5, 0x19, 0x0, 0x0, 0x0, 0x15f6fa0, 0xc4200f6e60, 0x1faffc0, 0xc4200f6f60)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/field.go:165 +0xc90
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.addFields(0x1faffc0, 0xc4200f6f60, 0xc420414500, 0x1, 0x2)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/field.go:199 +0x62
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.consoleEncoder.writeContext(0xc4204437c0, 0xc420601840, 0xc420414500, 0x1, 0x2)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/console_encoder.go:131 +0xcf
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.consoleEncoder.EncodeEntry(0xc4204437c0, 0x1, 0xbebef6c1af74e1ec, 0x4827658c562e, 0x1fe2ce0, 0x1714a5f, 0xe, 0x171b577, 0x16, 0x1, ...)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/console_encoder.go:110 +0x3de
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.(*ioCore).Write(0xc4203145d0, 0x1, 0xbebef6c1af74e1ec, 0x4827658c562e, 0x1fe2ce0, 0x1714a5f, 0xe, 0x171b577, 0x16, 0x1, ...)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/core.go:86 +0xa9
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc42007c580, 0xc420414500, 0x1, 0x2)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/entry.go:215 +0xe7
github.com/elastic/beats/vendor/go.uber.org/zap.(*SugaredLogger).log(0xc42000e190, 0xc420425001, 0x171b577, 0x16, 0x0, 0x0, 0x0, 0xc42004bdc8, 0x2, 0x2)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/sugar.go:234 +0xf6
github.com/elastic/beats/vendor/go.uber.org/zap.(*SugaredLogger).Warnw(0xc42000e190, 0x171b577, 0x16, 0xc42004bdc8, 0x2, 0x2)
	/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/sugar.go:185 +0x83
github.com/elastic/beats/libbeat/logp.(*Logger).Warnw(0xc42000e1a0, 0x171b577, 0x16, 0xc42004bdc8, 0x2, 0x2)
	/go/src/github.com/elastic/beats/libbeat/logp/logger.go:121 +0x60
github.com/elastic/beats/auditbeat/module/file_integrity.(*reader).consumeEvents(0xc420010210, 0xc4202a69c0)
	/go/src/github.com/elastic/beats/auditbeat/module/file_integrity/eventreader_fsnotify.go:85 +0x62d
created by github.com/elastic/beats/auditbeat/module/file_integrity.(*reader).Start
	/go/src/github.com/elastic/beats/auditbeat/module/file_integrity/eventreader_fsnotify.go:54 +0x3dd

I have several other instances of RHEL 6, Centos 6, Centos 7, Windows 10 and Windows Server 2008R2 with no issue.

Any ideas?

Thanks!
James


(Adrian Serrano) #2

Hi,

Which version of auditbeat is it?


(Jamesspi) #3

Hi @adrisr,

6.2.4.

Thanks!


(Adrian Serrano) #4

Can you share your auditbeat configuration?


(Jamesspi) #5

Sure, here is a slightly redacted version:

    auditbeat.modules:
- module: file_integrity
  paths:
  - /home
  - /bin
  - /usr/bin
  - /sbin
  - /usr/sbin
  - /etc
  exclude_files:
  - '(?i)\.sw[nop]$'
  - '~$'
  - '/\.git($|/)'
  scan_at_start: true
  scan_rate_per_sec: 50 MiB
  max_file_size: 5000 MiB
  hash_types: [sha256]
  recursive: true
setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 1
  index.codec: best_compression
output.elasticsearch:
  index: "XXXXXXXXXXXXXXXXXXX"
  proxy_url: XXXXXXXXXXXXXXXXXX
cloud.id: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
cloud.auth: XXXXXXXXXXX:XXXXXXXXXXX
xpack.monitoring.enabled: true
setup.template.name: "auditbeat"
setup.template.pattern: "auditbeat-*"
setup.dashboards.enabled: false
setup.dashboards.index: "auditbeat-*"

Thanks,
James


(Adrian Serrano) #6

Can you run auditbeat with debug mode (-d '*') and attach the lines above the crash?


(Jamesspi) #7

Sure, this is currently running in production so will need to raise a CR. Will keep you posted on progress.


(Jamesspi) #8

Hi @adrisr,

Here you go (slightly redacted again):

2018-06-12T10:00:11.462+0200	DEBUG	[file_integrity]	file_integrity/metricset.go:207	File changed since it was last seen	{"file_path": "/home/xxxxxx/xxxxxx/sedDk6pUG", "took": 10582, "event": {"old": null, "new": {"timestamp":"2018-06-12T08:00:11.457630097Z","path":"/home/xxxxxx/xxxxxx/sedDk6pUG","info":null,"source":"fsnotify","action":"attributes_modified"}}}
2018-06-12T10:00:11.463+0200	DEBUG	[file_integrity]	file_integrity/metricset.go:207	File changed since it was last seen	{"file_path": "/home/xxxx/xxxxx/sedDk6pUG", "took": 10913, "event": {"old": null, "new": {"timestamp":"2018-06-12T08:00:11.459856775Z","path":"/home/xxxxx/xxxxx/sedDk6pUG","info":null,"source":"fsnotify","action":"moved"}}}
2018-06-12T10:00:11.465+0200	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-06-12T08:00:11.457Z",
  "@metadata": {
    "beat": "auditbeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "beat": {
    "name": "xxxxx",
    "hostname": "xxxxx",
    "version": "6.2.4"
  },
  "event": {
    "module": "file_integrity",
    "action": [
      "attributes_modified"
    ]
  },
  "file": {
    "path": "/home/xxxxxx/xxxxx/sedDk6pUG"
  }
}
2018-06-12T10:00:11.465+0200	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-06-12T08:00:11.459Z",
  "@metadata": {
    "beat": "auditbeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "beat": {
    "name": "xxxxxx",
    "hostname": "xxxxxx",
    "version": "6.2.4"
  },
  "event": {
    "action": [
      "moved"
    ],
    "module": "file_integrity"
  },
  "file": {
    "path": "/home/xxxxx/xxxxx/sedDk6pUG"
  }
}
2018-06-12T10:00:11.470+0200	DEBUG	[module]	module/wrapper.go:177	Stopped metricSetWrapper[module=file_integrity, name=file, host=]
2018-06-12T10:00:11.470+0200	DEBUG	[module]	module/wrapper.go:120	Stopped Wrapper[name=file_integrity, len(metricSetWrappers)=1]
panic: runtime error: invalid memory address or nil pointer dereference

Thanks!
James


(Adrian Serrano) #9

While I still couldn't reproduce the issue, after some investigation I've found what could be the culprit.

Can you try this package to see if the problem is gone?

https://drive.google.com/file/d/1TArHDI3-GbZMm0BolxOBxorUACMJTLy_/view?usp=sharing


(Jamesspi) #10

Thanks @adrisr! I'll give this a go and keep you posted.

James


(Adrian Serrano) #11

Hi @jamesspi

Did you have the chance to test it?

If it fixed the problem we would like to release it asap


(Jamesspi) #12

Hi @adrisr,

Sorry, was away for a few days - raising a change to get this installed ASAP.

Thanks for following up!
James


(Adrian Serrano) #13

@jamesspi did it make a difference or still crashing?


(Jamesspi) #14

Hi Adrian,

Apologies for the late reply.

It didn’t happen again, so it looks like the fix worked :slight_smile:

Thanks a mill!

James


(Adrian Serrano) #15

@jamesspi can you tell me the exact version (and kernel version) for this RHEL6 box ? I'm still trying to reproduce it


(Jamesspi) #16

Hi Adrian,

Sorry for the slow responses, currently on vacation.

I’m getting the version for you through my ex colleague, as I am no longer with the same company. Will keep you posted.

Thanks!

James


(Adrian Serrano) #17

Hi @jamesspi,

You can give my contact details to your ex-coworker so I don't have to bother you anymore :slight_smile:

adrian.serrano at elastic.co