Var/audit logs messages include wierd characters

qnator · January 23, 2019, 3:54pm

Hello,

My current setup on my machine is:

Elasticsearch 6.5.4
Kibana 6.5.4
logstash 6.5.4

On a remote machine:
filebeat 6.5.4

The output for filebeat is logstash and its configured to pull for the var/audit folder. I'm able to see the var/audit/*.not_terminated logs but the message body contains weird characters.

How do I filter this out?

See below for an example of the garbled test:

f��@�J�(launchd::Audit recovery#)/var/audit/20190117194635.crash_recovery'�f9��@�J�(launchd::Audit startup'�9}�e@�Jyqasflags-am_success-am_failure$��'�}X��@�K@$��(begin evaluation'�Xv��@�KC$��(config.modify.com.apple.wifi(config.modify.'�v��@�KC$��(config.modify.com.apple.wifi(client /usr/libexec/airportd(creator /usr/libexec/airportd'��V��@�KC$��(end evaluation'�V}�e@�M�q0asflags-am_success-am_failure$��'�}}�e@�M�qasflags-am_success-am_failure$��'�}}�g@�M�qasflags-am_success-am_failure$��'�}X��@�N�$��a��b(begin evaluation'�Xt��@�N�$��a��b(system.login.console(system.login.console'�t��@�NH$��a��b(system.login.console( mechanism builtin:policy-banner'��@�O�$��\\��(EVerify password for record type Users 'admin' node '/Local/Default''��@�X$��\\��

steffens · January 24, 2019, 1:38pm

Are you using the beats input in logstash?

qnator · January 24, 2019, 9:00pm

Yes I am, see below for my 02-beats-input.conf file:

input {
beats {
port => 5044
ssl => false
}
}

The filter part of this file is commented out to indicate that it

is optional.

filter {
if [fileset][module] == "system" {
if [fileset][name] == "auth" {
grok {
match => { "message" => ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?",
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}",
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}",
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:[%{POSINT:[system][auth][pid]}])?: \s*%{DATA:[system][auth][user]} %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}",
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:[%{POSINT:[system][auth][pid]}])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}",
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:[%{POSINT:[system][auth][pid]}])?: new user: name=%{DATA:[system][auth][user][add][name]}, UID=%{NUMBER:[system][auth][user][add][uid]}, GID=%{NUMBER:[system][auth][user][add][gid]}, home=%{DATA:[system][auth][user][add][home]}, shell=%{DATA:[system][auth][user][add][shell]}$",
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:[%{POSINT:[system][auth][pid]}])?: %{GREEDYMULTILINE:[system][auth][message]}"] }
pattern_definitions => {
"GREEDYMULTILINE"=> "(.|\n)"
}
remove_field => "message"
}
date {
match => [ "[system][auth][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
geoip {
source => "[system][auth][ssh][ip]"
target => "[system][auth][ssh][geoip]"
}
}
else if [fileset][name] == "syslog" {
grok {
match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:[%{POSINT:[system][syslog][pid]}])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }
pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)" }
remove_field => "message"
}
date {
match => [ "[system][syslog][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
}

output {
elasticsearch {
hosts => "X.X.X.X:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}

steffens · January 25, 2019, 12:26pm

Have you tried to open the file locally? Is the file pure text? Which encoding does the original file use?

From system/auth manifest file:

    os.darwin:
      # this works in OS X < 10.8. Newer darwin versions don't write
      # ssh logs to files
      - /var/log/secure.log*

qnator · January 28, 2019, 4:19pm

I'm working on macOS High Sierra. I don't see a secure.log in the /var/log/ directory.

when I cat the var/audit/*.not_terminated log, I get the following below. How can I check the encoding of this file?

attribute = 'dsAttrTypeStandard:AuthenticationAuthority', value(s) = ';ShadowHash;HASHLIST:<SALTED-SHA512-PBKDF2,SRP-RFC5054-4096-SHA512-PBKDF2>,;Kerberosv5;;oadmin@LKDC:SHA1.2346091D099B7705A2515E4DE344ED3444AA1F4B;LKDC:SHA1.2346091D099B7705A2515E4DE344ED3444AA1F4B;,;SecureToken;''???
??\O
????=??=(EVerify password for record type Users 'oadmin' node '/Local/Default''??X ??\O ?L????4??5(begin evaluation'?Xx
??\O
?L$????4??5(system.privilege.admin(system.privilege.admin'?x?
??\O
?L$????4??5(system.privilege.admin(3client /usr/local/McAfee/AntiMalware/VShieldUpdate(4creator /usr/local/McAfee/AntiMalware/VShieldUpdate'??V
??\O
?L$????4??5(end evaluation'?VX
??\O
?]????4??5(begin evaluation'?Xx ??\O ?^????4??5(system.privilege.admin(system.privilege.admin'?x?
??\O
?^????4??5(system.privilege.admin(,client /usr/libexec/security_authtrampoline(4creator /usr/local/McAfee/AntiMalware/VShieldUpdate'??V ??\O ?^????4??5(end evaluation'?VX
??\O
??????4??5(begin evaluation'?Xx ??\O ??????4??5(system.privilege.admin(system.privilege.admin'?x?

qnator · January 28, 2019, 5:00pm

the command praudit cleans up the file:

header,86,11,SecSrvr AuthEngine,0,Mon Jan 28 09:04:29 2019, + 94 msec

subject,-1,root,wheel,root,wheel,52,100000,53,0.0.0.0

text,end evaluation

return,success,0

trailer,86

header,88,11,SecSrvr AuthEngine,0,Mon Jan 28 09:04:33 2019, + 398 msec

subject,-1,root,wheel,root,wheel,52,100000,53,0.0.0.0

text,begin evaluation

return,success,0

trailer,88

header,120,11,SecSrvr AuthEngine,0,Mon Jan 28 09:04:33 2019, + 399 msec

subject,-1,root,wheel,root,wheel,52,100000,53,0.0.0.0

text,system.privilege.admin

return,success,0

trailer,120

header,203,11,SecSrvr AuthEngine,0,Mon Jan 28 09:04:33 2019, + 399 msec

subject,-1,root,wheel,root,wheel,52,100000,53,0.0.0.0

text,system.privilege.admin

text,client /usr/local/McAfee/AntiMalware/VShieldUpdate

text,creator /usr/local/McAfee/AntiMalware/VShieldUpdate

return,success,0

trailer,203

header,86,11,SecSrvr AuthEngine,0,Mon Jan 28 09:04:33 2019, + 399 msec

subject,-1,root,wheel,root,wheel,52,100000,53,0.0.0.0

text,end evaluation

return,success,0

trailer,86

header,88,11,SecSrvr AuthEngine,0,Mon Jan 28 09:04:33 2019, + 418 msec

subject,-1,root,wheel,root,wheel,52,100000,53,0.0.0.0

text,begin evaluation

return,success,0

trailer,88

header,120,11,SecSrvr AuthEngine,0,Mon Jan 28 09:04:33 2019, + 419 msec

subject,-1,root,wheel,root,wheel,52,100000,53,0.0.0.0

text,system.privilege.admin

return,success,0

trailer,120

steffens · January 29, 2019, 6:34am

I'd say this confirms the file format actually being binary, not pure text.
Feel free to open an enhancement request for macOS audit logs: https://github.com/elastic/beats/issues/new/choose

system · February 26, 2019, 6:34am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.