Auditbeat fatal error (crash)

ppanagiotis · August 7, 2018, 10:08am

auditbeat occasionally crashes with the following error:

Aug 07 12:32:14 hostname auditbeat[10686]: fatal error: concurrent map writesA - Pastebin.com

The package is build with master branch (head commit b84e083625ec10)
OS: SMP Debian 4.9.88-1+deb9u1
Kernel: 4.9.0-6-amd64

ppanagiotis · August 7, 2018, 10:55am

As I can see commit 124c8a28f9 introduce the problem with that:

        for i := 0; i < numConsumers; i++ {
                go func() {
                        defer wg.Done()
                        for {
                                select {
                                case <-reporter.Done():
                                        return
                                case msgs := <-out:
                                        reporter.Event(buildMetricbeatEvent(msgs, ms.config))
                                }   
                        }   
                }() 
        }

numConsumers is equal with the number of the cpus (max = 4)

which is unsafe.
( if I set up max_procs: 1 at auditbeat.yml the problem goes away )

cc @adrisr

ppanagiotis · August 7, 2018, 3:14pm

As I can see It is fixed at go-libaudit

github.com

elastic/go-libaudit/blob/master/aucoalesce/id_lookup.go#L48


}


func (i *stringItem) isExpired() bool {
	return time.Now().After(i.timeout)
}


// UserCache is a cache of UID to username.
type UserCache struct {
	expiration time.Duration
	data       map[string]stringItem
	mutex      sync.Mutex
}


// NewUserCache returns a new UserCache. UserCache is not thread-safe.
func NewUserCache(expiration time.Duration) UserCache {
	return UserCache{
		expiration: expiration,
		data: map[string]stringItem{
			"0": {timeout: time.Unix(math.MaxInt64, 0), value: "root"},
		},
	}

I make a pull request https://github.com/elastic/beats/pull/7898

jsoriano · August 7, 2018, 4:00pm

Thanks!