### Author:  [@jamesspi](https://discuss.elastic.co/u/jamesspi)
#### Post date: [June 11, 2018, 10:25am UTC](https://discuss.elastic.co/t/auditbeat-randomly-shuts-down/135335/1 "2018-06-11T10:25:08Z")
Hi All,
I have an auditbeat instance running on RHEL 6, just the file integrity module. At certain times, it just stops running with the following error:
```
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xe606a5]
goroutine 82 [running]:
github.com/elastic/beats/vendor/github.com/joeshaw/multierror.(*MultiError).Error(0xc4200f6e00, 0x4, 0xc42a36980b)
/go/src/github.com/elastic/beats/vendor/github.com/joeshaw/multierror/multierror.go:43 +0x95
github.com/elastic/beats/vendor/github.com/pkg/errors.(*withMessage).Error(0xc4200f6e20, 0xad9977, 0x1a8200f6f60)
/go/src/github.com/elastic/beats/vendor/github.com/pkg/errors/errors.go:228 +0x37
github.com/elastic/beats/vendor/github.com/pkg/errors.(*withStack).Error(0xc4200f6e60, 0x20, 0x60000000020)
:1 +0x3c
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.encodeError(0x170bf0a, 0x5, 0x1f91640, 0xc4200f6e60, 0x1faffc0, 0xc4200f6f60, 0xc4200f6f40, 0xc4201bac00)
/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/error.go:46 +0x49
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.Field.AddTo(0x170bf0a, 0x5, 0x19, 0x0, 0x0, 0x0, 0x15f6fa0, 0xc4200f6e60, 0x1faffc0, 0xc4200f6f60)
/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/field.go:165 +0xc90
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.addFields(0x1faffc0, 0xc4200f6f60, 0xc420414500, 0x1, 0x2)
/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/field.go:199 +0x62
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.consoleEncoder.writeContext(0xc4204437c0, 0xc420601840, 0xc420414500, 0x1, 0x2)
/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/console_encoder.go:131 +0xcf
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.consoleEncoder.EncodeEntry(0xc4204437c0, 0x1, 0xbebef6c1af74e1ec, 0x4827658c562e, 0x1fe2ce0, 0x1714a5f, 0xe, 0x171b577, 0x16, 0x1, ...)
/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/console_encoder.go:110 +0x3de
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.(*ioCore).Write(0xc4203145d0, 0x1, 0xbebef6c1af74e1ec, 0x4827658c562e, 0x1fe2ce0, 0x1714a5f, 0xe, 0x171b577, 0x16, 0x1, ...)
/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/core.go:86 +0xa9
github.com/elastic/beats/vendor/go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc42007c580, 0xc420414500, 0x1, 0x2)
/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/zapcore/entry.go:215 +0xe7
github.com/elastic/beats/vendor/go.uber.org/zap.(*SugaredLogger).log(0xc42000e190, 0xc420425001, 0x171b577, 0x16, 0x0, 0x0, 0x0, 0xc42004bdc8, 0x2, 0x2)
/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/sugar.go:234 +0xf6
github.com/elastic/beats/vendor/go.uber.org/zap.(*SugaredLogger).Warnw(0xc42000e190, 0x171b577, 0x16, 0xc42004bdc8, 0x2, 0x2)
/go/src/github.com/elastic/beats/vendor/go.uber.org/zap/sugar.go:185 +0x83
github.com/elastic/beats/libbeat/logp.(*Logger).Warnw(0xc42000e1a0, 0x171b577, 0x16, 0xc42004bdc8, 0x2, 0x2)
/go/src/github.com/elastic/beats/libbeat/logp/logger.go:121 +0x60
github.com/elastic/beats/auditbeat/module/file_integrity.(*reader).consumeEvents(0xc420010210, 0xc4202a69c0)
/go/src/github.com/elastic/beats/auditbeat/module/file_integrity/eventreader_fsnotify.go:85 +0x62d
created by github.com/elastic/beats/auditbeat/module/file_integrity.(*reader).Start
/go/src/github.com/elastic/beats/auditbeat/module/file_integrity/eventreader_fsnotify.go:54 +0x3dd
```
I have several other instances of RHEL 6, Centos 6, Centos 7, Windows 10 and Windows Server 2008R2 with no issue.
Any ideas?
Thanks!
James
---
### Author:  [@adrisr](https://discuss.elastic.co/u/adrisr)
#### Post date: [June 11, 2018, 1:53pm UTC](https://discuss.elastic.co/t/auditbeat-randomly-shuts-down/135335/2 "2018-06-11T13:53:45Z")
Hi,
Which version of auditbeat is it?
---
### Author:  [@jamesspi](https://discuss.elastic.co/u/jamesspi)
#### Post date: [June 11, 2018, 2:23pm UTC](https://discuss.elastic.co/t/auditbeat-randomly-shuts-down/135335/3 "2018-06-11T14:23:31Z")
Hi @adrisr,
6.2.4.
Thanks!
---
### Author:  [@adrisr](https://discuss.elastic.co/u/adrisr)
#### Post date: [June 11, 2018, 3:08pm UTC](https://discuss.elastic.co/t/auditbeat-randomly-shuts-down/135335/4 "2018-06-11T15:08:09Z")
> [@jamesspi](#):
>
> eventreader\_fsnotify.go:85
Can you share your auditbeat configuration?
---
### Author:  [@jamesspi](https://discuss.elastic.co/u/jamesspi)
#### Post date: [June 11, 2018, 3:10pm UTC](https://discuss.elastic.co/t/auditbeat-randomly-shuts-down/135335/5 "2018-06-11T15:10:08Z")
Sure, here is a slightly redacted version:
```
auditbeat.modules:
- module: file_integrity
paths:
- /home
- /bin
- /usr/bin
- /sbin
- /usr/sbin
- /etc
exclude_files:
- '(?i)\.sw[nop]$'
- '~$'
- '/\.git($|/)'
scan_at_start: true
scan_rate_per_sec: 50 MiB
max_file_size: 5000 MiB
hash_types: [sha256]
recursive: true
setup.template.settings:
index.number_of_shards: 1
index.number_of_replicas: 1
index.codec: best_compression
output.elasticsearch:
index: "XXXXXXXXXXXXXXXXXXX"
proxy_url: XXXXXXXXXXXXXXXXXX
cloud.id: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
cloud.auth: XXXXXXXXXXX:XXXXXXXXXXX
xpack.monitoring.enabled: true
setup.template.name: "auditbeat"
setup.template.pattern: "auditbeat-*"
setup.dashboards.enabled: false
setup.dashboards.index: "auditbeat-*"
```
Thanks,
James
---
### Author:  [@adrisr](https://discuss.elastic.co/u/adrisr)
#### Post date: [June 12, 2018, 7:14am UTC](https://discuss.elastic.co/t/auditbeat-randomly-shuts-down/135335/6 "2018-06-12T07:14:00Z")
Can you run auditbeat with debug mode (`-d '*'`) and attach the lines above the crash?
---
### Author:  [@jamesspi](https://discuss.elastic.co/u/jamesspi)
#### Post date: [June 12, 2018, 7:15am UTC](https://discuss.elastic.co/t/auditbeat-randomly-shuts-down/135335/7 "2018-06-12T07:15:14Z")
Sure, this is currently running in production so will need to raise a CR. Will keep you posted on progress.
---
### Author:  [@jamesspi](https://discuss.elastic.co/u/jamesspi)
#### Post date: [June 12, 2018, 1:49pm UTC](https://discuss.elastic.co/t/auditbeat-randomly-shuts-down/135335/8 "2018-06-12T13:49:16Z")
Hi @adrisr,
Here you go (slightly redacted again):
```
2018-06-12T10:00:11.462+0200 DEBUG [file_integrity] file_integrity/metricset.go:207 File changed since it was last seen {"file_path": "/home/xxxxxx/xxxxxx/sedDk6pUG", "took": 10582, "event": {"old": null, "new": {"timestamp":"2018-06-12T08:00:11.457630097Z","path":"/home/xxxxxx/xxxxxx/sedDk6pUG","info":null,"source":"fsnotify","action":"attributes_modified"}}}
2018-06-12T10:00:11.463+0200 DEBUG [file_integrity] file_integrity/metricset.go:207 File changed since it was last seen {"file_path": "/home/xxxx/xxxxx/sedDk6pUG", "took": 10913, "event": {"old": null, "new": {"timestamp":"2018-06-12T08:00:11.459856775Z","path":"/home/xxxxx/xxxxx/sedDk6pUG","info":null,"source":"fsnotify","action":"moved"}}}
2018-06-12T10:00:11.465+0200 DEBUG [publish] pipeline/processor.go:275 Publish event: {
"@timestamp": "2018-06-12T08:00:11.457Z",
"@metadata": {
"beat": "auditbeat",
"type": "doc",
"version": "6.2.4"
},
"beat": {
"name": "xxxxx",
"hostname": "xxxxx",
"version": "6.2.4"
},
"event": {
"module": "file_integrity",
"action": [
"attributes_modified"
]
},
"file": {
"path": "/home/xxxxxx/xxxxx/sedDk6pUG"
}
}
2018-06-12T10:00:11.465+0200 DEBUG [publish] pipeline/processor.go:275 Publish event: {
"@timestamp": "2018-06-12T08:00:11.459Z",
"@metadata": {
"beat": "auditbeat",
"type": "doc",
"version": "6.2.4"
},
"beat": {
"name": "xxxxxx",
"hostname": "xxxxxx",
"version": "6.2.4"
},
"event": {
"action": [
"moved"
],
"module": "file_integrity"
},
"file": {
"path": "/home/xxxxx/xxxxx/sedDk6pUG"
}
}
2018-06-12T10:00:11.470+0200 DEBUG [module] module/wrapper.go:177 Stopped metricSetWrapper[module=file_integrity, name=file, host=]
2018-06-12T10:00:11.470+0200 DEBUG [module] module/wrapper.go:120 Stopped Wrapper[name=file_integrity, len(metricSetWrappers)=1]
panic: runtime error: invalid memory address or nil pointer dereference
```
Thanks!
James
---
### Author:  [@adrisr](https://discuss.elastic.co/u/adrisr)
#### Post date: [June 13, 2018, 2:21am UTC](https://discuss.elastic.co/t/auditbeat-randomly-shuts-down/135335/9 "2018-06-13T02:21:52Z")
While I still couldn't reproduce the issue, after some investigation I've found what could be the culprit.
Can you try this package to see if the problem is gone?
[https://drive.google.com/file/d/1TArHDI3-GbZMm0BolxOBxorUACMJTLy\_/view?usp=sharing](https://drive.google.com/file/d/1TArHDI3-GbZMm0BolxOBxorUACMJTLy_/view?usp=sharing)
---
### Author:  [@jamesspi](https://discuss.elastic.co/u/jamesspi)
#### Post date: [June 13, 2018, 6:54am UTC](https://discuss.elastic.co/t/auditbeat-randomly-shuts-down/135335/10 "2018-06-13T06:54:09Z")
Thanks @adrisr! I'll give this a go and keep you posted.
James
---
### Author:  [@adrisr](https://discuss.elastic.co/u/adrisr)
#### Post date: [June 19, 2018, 4:37am UTC](https://discuss.elastic.co/t/auditbeat-randomly-shuts-down/135335/11 "2018-06-19T04:37:38Z")
Hi @jamesspi
Did you have the chance to test it?
If it fixed the problem we would like to release it asap
---
### Author:  [@jamesspi](https://discuss.elastic.co/u/jamesspi)
#### Post date: [June 19, 2018, 7:08am UTC](https://discuss.elastic.co/t/auditbeat-randomly-shuts-down/135335/12 "2018-06-19T07:08:00Z")
Hi @adrisr,
Sorry, was away for a few days - raising a change to get this installed ASAP.
Thanks for following up!
James
---
### Author:  [@adrisr](https://discuss.elastic.co/u/adrisr)
#### Post date: [June 27, 2018, 8:34am UTC](https://discuss.elastic.co/t/auditbeat-randomly-shuts-down/135335/13 "2018-06-27T08:34:49Z")
@jamesspi did it make a difference or still crashing?
---
### Author:  [@jamesspi](https://discuss.elastic.co/u/jamesspi)
#### Post date: [June 28, 2018, 3:29pm UTC](https://discuss.elastic.co/t/auditbeat-randomly-shuts-down/135335/14 "2018-06-28T15:29:35Z")
Hi Adrian,
Apologies for the late reply.
It didn’t happen again, so it looks like the fix worked 🙂
Thanks a mill!
James
---
### Author:  [@adrisr](https://discuss.elastic.co/u/adrisr)
#### Post date: [July 2, 2018, 7:01pm UTC](https://discuss.elastic.co/t/auditbeat-randomly-shuts-down/135335/15 "2018-07-02T19:01:45Z")
@jamesspi can you tell me the exact version (and kernel version) for this RHEL6 box ? I'm still trying to reproduce it
---
### Author:  [@jamesspi](https://discuss.elastic.co/u/jamesspi)
#### Post date: [July 4, 2018, 5:31pm UTC](https://discuss.elastic.co/t/auditbeat-randomly-shuts-down/135335/16 "2018-07-04T17:31:03Z")
Hi Adrian,
Sorry for the slow responses, currently on vacation.
I’m getting the version for you through my ex colleague, as I am no longer with the same company. Will keep you posted.
Thanks!
James
---
### Author:  [@adrisr](https://discuss.elastic.co/u/adrisr)
#### Post date: [July 4, 2018, 6:26pm UTC](https://discuss.elastic.co/t/auditbeat-randomly-shuts-down/135335/17 "2018-07-04T18:26:54Z")
Hi @jamesspi,
You can give my contact details to your ex-coworker so I don't have to bother you anymore 🙂
adrian.serrano at [elastic.co](http://elastic.co)
---
### Author:  [@Salty](https://discuss.elastic.co/u/Salty)
#### Post date: [May 1, 2019, 5:10pm UTC](https://discuss.elastic.co/t/auditbeat-randomly-shuts-down/135335/18 "2019-05-01T17:10:24Z")
Hi,
Just ran into this same issue.
Host: CentOS 6.10
Kernel: 2.6.32-754.12.1.el6.x86\_64
I've tried Auditbeat v6.7.1 and v7.0.1 and they both randomly shutdown.
I tried the version you posted (v6.2.4\_SNAPSHOT-1) and it hasn't crashed yet. 👍
Is there anything I can do to help identify/test/verify the crash so the issue can be resolved with the newer clients?
Thanks
---
### Author:  [@adrisr](https://discuss.elastic.co/u/adrisr)
#### Post date: [May 2, 2019, 10:20am UTC](https://discuss.elastic.co/t/auditbeat-randomly-shuts-down/135335/19 "2019-05-02T10:20:44Z")
The fix for this issue ([#7753](https://github.com/elastic/beats/issues/7753)) is already merged into 6.7 / 7.0. You must be experiencing a different bug that is not present in 6.2.
Can you see any stack trace in the logs like the one in this thread?
Also share your auditbeat.yml contents.
---
### Author:  [@Salty](https://discuss.elastic.co/u/Salty)
#### Post date: [May 2, 2019, 3:34pm UTC](https://discuss.elastic.co/t/auditbeat-randomly-shuts-down/135335/20 "2019-05-02T15:34:46Z")
Hi,
I created a new CentOS 6.10 VM this morning and installed Auditbeat v6.7.1.
I upgraded the kernel but everything else is as CentOS shipped it.
Auditbeat.yml - pretty much a stock file.
# This is an example configuration file highlighting only the most common
# options. The auditbeat.reference.yml file from the same directory contains all
# the supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# [https://www.elastic.co/guide/en/beats/auditbeat/index.html](https://www.elastic.co/guide/en/beats/auditbeat/index.html)
#========================== Modules configuration =============================
auditbeat.modules:
- module: auditd
- module: file\_integrity
paths:
- module: system
datasets:
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number\_of\_shards: 3
#index.codec: best\_compression
#\_source.enabled: false
#================================ General =====================================
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:
# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]
# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging
#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here, or by using the `-setup` CLI flag or the `setup` command.
#setup.dashboards.enabled: false
# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the [artifacts.elastic.co](http://artifacts.elastic.co)
# website.
#setup.dashboards.url:
#============================== Kibana =====================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: [http://localhost:5601/path](http://localhost:5601/path)
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
#host: "localhost:5601"
# Kibana Space ID
# ID of the Kibana Space into which the dashboards should be loaded. By default,
# the Default Space will be used.
#space.id:
#============================= Elastic Cloud ==================================
# These settings simplify using auditbeat with the Elastic Cloud ([https://cloud.elastic.co/](https://cloud.elastic.co/)).
# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:
# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `:`.
#cloud.auth:
#================================ Outputs =====================================
# Configure what output to use when sending the data collected by the beat.
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["192.168.153.177:9200"]
# Enabled ilm (beta) to use index lifecycle management instead daily indices.
#ilm.enabled: false
# Optional protocol and basic auth credentials.
#protocol: "https"
#username: "elastic"
#password: "changeme"
#----------------------------- Logstash output --------------------------------
#output.logstash:
# The Logstash hosts
#hosts: ["localhost:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate\_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
#================================ Processors =====================================
# Configure processors to enhance or manipulate events generated by the beat.
processors:
- add\_host\_metadata: ~
- add\_cloud\_metadata: ~
#================================ Logging =====================================
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["\*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["\*"]
[Next page](https://discuss.elastic.co/t/auditbeat-randomly-shuts-down/135335.md?page=2)