Thought I'd better start this as a separate topic. At 1am yesterday morning, the messages from Auditbeat stopped appearing in ELK. No changes were made to either Auditbeat or Logstash filtering. Just stopped dead. I was told to check the Auditbeat Log for a particular message (Second one below) as there could be a system limit that has been hit. Trouble is, I don't have any idea what to check. Any help gratefully received.
|2018-04-19T16:39:36.870+0100|INFO|[auditd]|auditd/audit_linux.go:192|audit status from kernel at start|{"audit_status": {"Mask":892548912,"Enabled":1,"Failure":1,"PID":0,"RateLimit":0,"BacklogLimit":8196,"Lost":3446808471,"Backlog":0,"FeatureBitmap":0,"BacklogWaitTime":0}}|
|---|---|---|---|---|---|
|2018-04-19T16:40:04.236+0100|INFO|[monitoring]|log/log.go:124|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"auditd":{"lost":293},"beat":{"cpu":{"system":{"ticks":17180,"time":17188},"total":{"ticks":46220,"time":46228,"value":46220},"user":{"ticks":29040,"time":29040}},"info":{"ephemeral_id":"c853ae42-1c9b-49b6-8416-b06b174d331d","uptime":{"ms":30041}},"memstats":{"gc_next":9708848,"memory_alloc":8230792,"memory_total":762385304,"rss":34197504}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"acked":23511,"batches":26,"total":23511},"read":{"bytes":156},"type":"logstash","write":{"bytes":3366714}},"pipeline":{"clients":1,"events":{"active":559,"published":24070,"retry":864,"total":24070},"queue":{"acked":23511}}},"metricbeat":{"auditd":{"auditd":{"events":24073,"success":24073}}},"system":{"cpu":{"cores":38},"load":{"1":0.84,"15":0.24,"5":0.37,"norm":{"1":0.0221,"15":0.0063,"5":0.0097}}}}}}|
I have checked the Auditbeat config and output and they both test OK. I tried remove anything clever from Auditbeat config and logstash filtering and still not receiving any messages.