Filebeat Processing /var/log/secure

[aviator]

Hi

ES 7.0.2
Am trying to view the Filebeat System New user and groups dashboard but the results are sporadic at best.
Filebeat 7.2.0 installed locally on ES instance and on remote machine to confirm. Configured to harvest /var/log/secure
Add a user and watch in Discovery and the event(s) get shipped quickly (filebeat -e -d "*") but the event may or may not turn up either in the dashboard or Discovery. Maybe an hour later but sometimes nothing.
Clean install with no load on the machine (this is the only input configured to check performance)
Ouput of

GET filebeat-*/_search?q=exists:system.auth.useradd

shows the document arrives within a few seconds but again the Discovery does not reflect this.

Is there a default post process slowing things down before it becomes available? The syslog message log seems to be available without any delay just the auth.

Regards

Ed

[rugenl (Len Rugen)]

Check the @timestamp of the event vs that of the log, it could be a timezone interpretation problem.

[aviator]

Timestamps look to be in sync, perhaps spoke too soon on the ingest being ok, running the filebeat debug I see the two documents being created for an adduser command. They are being sent to

filebeat-7.2.0-system-auth-pipeline

A check of the stats using

GET _nodes/stats/ingest

Shows the following under the auth pipeline (two below are the only failed entries)

"geoip" : {
"count" : 491,
"time_in_millis" : 287,
"current" : 0,
"failed" : 366
}
},
{
"script" : {
"count" : 491,
"time_in_millis" : 27,
"current" : 0,
"failed" : 366

The previous count was 489 and failed 364 so the two documents being ingested are not getting processed correctly?

Where could I look for further troubleshooting?

Regards

Ed

[rugenl (Len Rugen)]

Group fails are common, usually private ips. That won't fail ingest, just won't populate fields.

[rugenl (Len Rugen)]

Geoip, spellfix....

[aviator]

This is a new install - previously used the elastic geoip module so went ahead and followed this:

Enrich events with geoIP information | Filebeat Reference [7.2] | Elastic

have stared at this issue for hours and probably cannot see the obvious - is the geoip enrichment for the auth pipeline named something other than geoip?

Regards

Ed