FreeIPA Log Configuration for Logstash (Grok Filter)

Hello,

I was trying to follow https://www.freeipa.org/page/Centralized_Logging to configure the following logs from FreeIPA in logstash:

/var/log/httpd/error_log: FreeIPA API call logs (and Apache errors)
/var/log/krb5kdc.log: FreeIPA KDC utilization
/var/log/dirsrv/slapd-$REALM/access: Directory Server utilization
/var/log/dirsrv/slapd-$REALM/errors: Directory Server errors (including mentioned replication errors)

The issue is the page is for rsyslog but I'm using the following setup:

Filebeat > Redis > Logstash > Elasticsearch

Would anyone have any recommendations on how to create a grok filter for the above logs using the above setup? Thank you for any and all help!

Thank You

What do the logs look like?

Hello,

Here is a sample of data for each log.

/var/log/httpd/error_log:

[Sun Jun 18 03:41:01.802587 2017] [auth_digest:notice] [pid 17530] AH01757: generating secret for digest authentication ...
[Sun Jun 18 03:41:01.803553 2017] [lbmethod_heartbeat:notice] [pid 17530] AH02282: No slotmem from mod_heartmonitor
[Sun Jun 18 03:41:01.803571 2017] [:warn] [pid 17530] NSSSessionCacheTimeout is deprecated. Ignoring.
[Sun Jun 18 03:41:01.807129 2017] [mpm_prefork:notice] [pid 17530] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) mod_auth_gssapi/1.4.0 mod_nss/1.0.14 NSS/3.21 Basic ECC mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Sun Jun 18 03:41:01.807148 2017] [core:notice] [pid 17530] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Sun Jun 18 03:41:06.643044 2017] [:error] [pid 30351] ipa: INFO: *** PROCESS START ***
[Sun Jun 18 03:41:06.646679 2017] [:error] [pid 30350] ipa: INFO: *** PROCESS START ***

/var/log/krb5kdc.log:

Jun 19 08:19:23 sipa01.c01.aaa.gen.nyc.aevtech.net krb5kdc17497: closing down fd 4
Jun 19 08:19:23 sipa01.c01.aaa.gen.nyc.aevtech.net krb5kdc17497: TGS_REQ (6 etypes {18 17 16 23 25 26}) 10.34.80.90: ISSUE: authtime 1497860363, etypes {rep=18 tkt=18 ses=18}, host/seg04.c20.streaming.nyc.aevtech.net@AEVTECH.NET for ldap/sipa01.c01.aaa.gen.nyc.aevtech.net@AEVTECH.NET
Jun 19 08:19:23 sipa01.c01.aaa.gen.nyc.aevtech.net krb5kdc17497: closing down fd 4

/var/log/dirsrv/slapd-MLBAM-NET/access:

[19/Jun/2017:08:22:10.256710575 +0000] conn=146207 op=536114 RESULT err=0 tag=101 nentries=0 etime=0
[19/Jun/2017:08:22:10.258331605 +0000] conn=252725 op=1 UNBIND
[19/Jun/2017:08:22:10.258345573 +0000] conn=252725 op=1 fd=305 closed - U1
[19/Jun/2017:08:22:10.259720144 +0000] conn=252726 fd=305 slot=305 connection from 10.34.116.95 to 10.34.192.40
[19/Jun/2017:08:22:10.260070526 +0000] conn=252726 op=0 SRCH base="" scope=0 filter="(objectClass=)" attrs=" altServer namingContexts supportedControl supportedExtension supportedFeatures supportedLDAPVersion supportedSASLMechanisms domaincontrollerfunctionality defaultnamingcontext lastusn highestcommittedusn aci"
[19/Jun/2017:08:22:10.260958407 +0000] conn=252726 op=0 RESULT err=0 tag=101 nentries=1 etime=0
[19/Jun/2017:08:22:10.268216523 +0000] conn=146207 op=536115 SRCH base="dc=aevtech,dc=net" scope=2 filter="(&(|(objectClass=krbprincipalaux)(objectClass=krbprincipal)(objectClass=ipakrbprincipal))(|(ipaKrbPrincipalAlias=host/java05.ad01.hls.aevtech.net@AEVTECH.NET)(krbPrincipalName:caseIgnoreIA5Match:=host/java05.ad01.hls.nyc.aevtech.net@AEVTECH.NET)))" attrs="krbPrincipalName krbCanonicalName krbUPEnabled krbPrincipalKey krbTicketPolicyReference krbPrincipalExpiration krbPasswordExpiration krbPwdPolicyReference krbPrincipalType krbPwdHistory krbLastPwdChange krbPrincipalAliases krbLastSuccessfulAuth krbLastFailedAuth krbLoginFailedCount krbPrincipalAuthInd krbExtraData krbLastAdminUnlock krbObjectReferences krbTicketFlags krbMaxTicketLife krbMaxRenewableAge nsAccountLock passwordHistory ipaKrbAuthzData ipaUserAuthType ipatokenRadiusConfigLink objectClass"

/var/log/dirsrv/slapd-MLBAM-NET/errors:

[15/Jun/2017:19:28:34.149592432 +0000] Retry count exceeded in delete
[15/Jun/2017:19:28:34.172819252 +0000] DSRetroclPlugin - delete_changerecord: could not delete change record 277064 (rc: 51)
[15/Jun/2017:19:28:46.443457661 +0000] schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=mlbam,dc=net
[15/Jun/2017:19:28:46.447371413 +0000] schema-compat-plugin - Finished plugin initialization.
[16/Jun/2017:18:28:20.827952664 +0000] is_allowed_to_access_attr - [file ipa_pwd_extop.c, line 777]: slapi_access_allowed does not allow WRITE to ipaProtectedOperation;write_keys!
[16/Jun/2017:18:28:20.830961454 +0000] ipapwd_getkeytab - [file ipa_pwd_extop.c, line 1651]: Not allowed to set keytab on [host/hyp189..nyc.aevtech.net@AEVTECH.NET]!
[16/Jun/2017:18:28:22.701435694 +0000] is_allowed_to_access_attr - [file ipa_pwd_extop.c, line 777]: slapi_access_allowed does not allow WRITE to ipaProtectedOperation;write_keys!
[16/Jun/2017:18:28:22.704530887 +0000] ipapwd_getkeytab - [file ipa_pwd_extop.c, line 1651]: Not allowed to set keytab on [host/hyp191.nyc.aevtech.net@AEVTECH.NET]!

Let me know if you need anything else so I can send it to you as soon as possible.

Thank you!

Ok, so what groks have you tried for each?

Hello,

I was going to try the following:

grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}

I wasn't sure if that would work with these logs. I'm also new to using logstash and grok filters so I wasn't 100% sure if this would work from what I read since the article was based on rsyslog.

Thank You

Hello,

Would the above grok filter work? I am new to grok filters and I'm trying to understand how to structure them depending on the log output. My confusion is when it comes to the "match" field in the grok filter, how do we know which options to select for the match field?

Thank you

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.