Filebeat Processing /var/log/secure


ES 7.0.2
Am trying to view the Filebeat System New user and groups dashboard but the results are sporadic at best.
Filebeat 7.2.0 installed locally on ES instance and on remote machine to confirm. Configured to harvest /var/log/secure
Add a user and watch in Discovery and the event(s) get shipped quickly (filebeat -e -d "*") but the event may or may not turn up either in the dashboard or Discovery. Maybe an hour later but sometimes nothing.
Clean install with no load on the machine (this is the only input configured to check performance)
Ouput of

GET filebeat-*/_search?q=exists:system.auth.useradd

shows the document arrives within a few seconds but again the Discovery does not reflect this.

Is there a default post process slowing things down before it becomes available? The syslog message log seems to be available without any delay just the auth.



Check the @timestamp of the event vs that of the log, it could be a timezone interpretation problem.

Timestamps look to be in sync, perhaps spoke too soon on the ingest being ok, running the filebeat debug I see the two documents being created for an adduser command. They are being sent to


A check of the stats using

GET _nodes/stats/ingest

Shows the following under the auth pipeline (two below are the only failed entries)

"geoip" : {
"count" : 491,
"time_in_millis" : 287,
"current" : 0,
"failed" : 366
"script" : {
"count" : 491,
"time_in_millis" : 27,
"current" : 0,
"failed" : 366

The previous count was 489 and failed 364 so the two documents being ingested are not getting processed correctly?

Where could I look for further troubleshooting?



Group fails are common, usually private ips. That won't fail ingest, just won't populate fields.

Geoip, spellfix....

This is a new install - previously used the elastic geoip module so went ahead and followed this:

have stared at this issue for hours and probably cannot see the obvious - is the geoip enrichment for the auth pipeline named something other than geoip?



This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.