Audit Log Logstash Conf not working

Hi,

I am trying to parse the audit log but I am not able to.

This is my conf.

input {
  file {
    path => "/var/log/audit/audit.log"
    start_position => "beginning"
  }
}
filter {
  grok {
        match => { "message" => "type=%{WORD:audit_type} msg=audit\(%{GREEDYDATA:audit_epoch}\): pid=%{NUMBER:audit_pid} uid=%{NUMBER:audit_uid} auid=%{NUMBER:audit_audid} ses=%{NUMBER:audit_ses} subj=%{GREEDYDATA:audit_subject} msg='%{GREEDYDATA:audit_message}'" }
        add_tag => "selinux_audit"
  }
}
output {
    rabbitmq {
        exchange => "elasticsearch-exchange"
        exchange_type => "direct"
        key => "logstash-routing_key"
        host => "localhost"
        vhost => "es_vhost"
        durable => true
        persistent => true
        port => 5672
        user => "admin"
        password => "mqadmin"
}
stdout {
    codec => rubydebug
  }
}

This is the error I am getting.

{:timestamp=>"2017-11-29T05:40:45.087000-0500", :message=>"fetched an invalid config", :config=>"#

You've truncated the error message. Please show all of it.

{:timestamp=>"2017-11-29T05:40:45.087000-0500", :message=>"fetched an invalid config", :config=>"# LogStash config file for RabbitMQ\n# From: https://confluence.oceanobservatories.org/display/CIDev/Logging+--+logstash+configuration\n\ninput {\n file {\n type => \"selinux_audit\"\n path => \"/var/log/audit/audit.log\"\n }\n}\n\nfilter {\n grok {\n\tmatch => { \"pattern\" => \"type=%{WORD:audit_type} msg=audit\\(%{GREEDYDATA:audit_epoch}\\): pid=%{NUMBER:audit_pid} uid=%{NUMBER:audit_uid} auid=%{NUMBER:audit_audid} ses=%{NUMBER:audit_ses} subj=%{GREEDYDATA:audit_subject} msg='%{GREEDYDATA:audit_message}'\" } \t\n\tadd_tag => \"selinux_audit\" \n }\n}\noutput {\n rabbitmq {\n exchange => \"elasticsearch-exchange\"\n exchange_type => \"direct\"\n key => \"logstash-routing_key\"\n host => \"localhost\"\n vhost => \"es_vhost\"\n durable => true\n persistent => true\n port => 5672\n user => \"admin\"\n password => \"mqadmin\" \n}\n}\n\ntype=DAEMON_START msg=audit(1508743577.175:1298): op=start ver=2.6.5 format=raw kernel=3.10.0-514.21.2.el7.x86_64 auid=4294967295 pid=378 subj=system_u:system_r:auditd_t:s0 res=success\ntype=LOGIN msg=audit(1508743592.409:60): pid=5808 uid=0 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 old-auid=4294967295 auid=1000 old-ses=4294967295 ses=1 res=1\ntype=USER_CMD msg=audit(1508743592.893:81): pid=5916 uid=1000 auid=1000 ses=1 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='cwd=\"/home/ec2-user\" cmd=63686D6F64202B78202F746D702F7363726970745F333433362E7368 terminal=? res=success'\ntype=USER_CMD msg=audit(1508743592.900:86): pid=5918 uid=1000 auid=1000 ses=1 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='cwd=\"/home/ec2-user\" cmd=\"/tmp/script_3436.sh\" terminal=? res=success'\ntype=USER_CMD msg=audit(1508743593.726:89): pid=6490 uid=0 auid=1000 ses=1 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='cwd=\"/home/ec2-user\" cmd=79756D202D7920696E7374616C6C207075707065742D6167656E742D312E332E32 terminal=? res=success'\ntype=USER_CMD msg=audit(1508743639.378:99): pid=9450 uid=0 auid=1000 ses=1 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='cwd=\"/home/ec2-user\" cmd=79756D202D7920696E7374616C6C207275627967656D732D322E302E31342E31 terminal=? res=success'\ntype=DAEMON_END msg=audit(1508743694.905:1299): op=terminate auid=0 pid=1 subj=system_u:system_r:init_t:s0 res=success\ntype=DAEMON_START msg=audit(1511852730.968:9416): op=start ver=2.6.5 format=raw kernel=3.10.0-514.21.2.el7.x86_64 auid=4294967295 pid=437 subj=system_u:system_r:auditd_t:s0 res=success\ntype=LOGIN msg=audit(1511853443.426:52): pid=2083 uid=0 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 old-auid=4294967295 auid=1000 old-ses=4294967295 ses=1 res=1\ntype=USER_CMD msg=audit(1511853447.608:63): pid=2108 uid=1000 auid=1000 ses=1 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='cwd=\"/home/ec2-user\" cmd=7375202D terminal=pts/0 res=success'\ntype=LOGIN msg=audit(1511856061.588:78): pid=2230 uid=0 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 old-auid=4294967295 auid=0 old-ses=4294967295 ses=2 res=1\ntype=LOGIN msg=audit(1511859661.738:107): pid=9396 uid=0 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 old-auid=4294967295 auid=0 old-ses=4294967295 ses=3 res=1\ntype=LOGIN msg=audit(1511863190.680:145): pid=9545 uid=0 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 old-auid=4294967295 auid=1000 old-ses=4294967295 ses=4 res=1\ntype=USER_CMD msg=audit(1511863194.163:156): pid=9570 uid=1000 auid=1000 ses=4 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='cwd=\"/home/ec2-user\" cmd=7375202D terminal=pts/1 res=success'\ntype=LOGIN msg=audit(1511863261.783:165): pid=9597 uid=0 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 old-auid=4294967295 auid=0 old-ses=4294967295 ses=5 res=1\ntype=LOGIN msg=audit(1511866861.835:184): pid=10308 uid=0 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 old-auid=4294967295 auid=0 old-ses=4294967295 ses=6 res=1\ntype=MAC_CONFIG_CHANGE msg=audit(1511868204.577:215): bool=nis_enabled val=1 old_val=0 auid=1000 ses=1\ntype=MAC_POLICY_LOAD msg=audit(1511868204.948:216): policy loaded auid=1000 ses=1\ntype=AVC msg=audit(1511868222.820:233): avc: denied { getattr } for pid=11030 comm=\"sh\" path=\"/usr/bin/systemd-notify\" dev=\"xvda2\" ino=12727806 scontext=system_u:system_r:rabbitmq_t:s0 tcontext=system_u:object_r:systemd_notify_exec_t:s0 tclass=file\ntype=AVC msg=audit(1511868222.820:234): avc: denied { getattr } for pid=11030 comm=\"sh\" path=\"/usr/bin/systemd-notify\" dev=\"xvda2\" ino=12727806 scontext=system_u:system_r:rabbitmq_t:s0 tcontext=system_u:object_r:systemd_notify_exec_t:s0 tclass=file\ntype=NETFILTER_CFG msg=audit(1511868759.569:264): table=filter family=2 entries=0\ntype=NETFILTER_CFG msg=audit(1511868759.570:265): table=filter family=2 entries=4\ntype=LOGIN msg=audit(1511870461.886:312): pid=12277 uid=0 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 old-auid=4294967295 auid=0 old-ses=4294967295 ses=7 res=1\ntype=LOGIN msg=audit(1511874061.936:344): pid=13186 uid=0 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 old-auid=4294967295 auid=0 old-ses=4294967295 ses=8 res=1\ntype=LOGIN msg=audit(1511877661.984:393): pid=14360 uid=0 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 old-auid=4294967295 auid=0 old-ses=4294967295 ses=9 res=1\ntype=DAEMON_END msg=audit(1511879173.667:9417): op=terminate auid=0 pid=1 subj=system_u:system_r:init_t:s0 res=success\ntype=DAEMON_START msg=audit(1511932835.502:8267): op=start ver=2.6.5 format=raw kernel=3.10.0-514.21.2.el7.x86_64 auid=4294967295 pid=391 subj=system_u:system_r:auditd_t:s0 res=success\ntype=AVC msg=audit(1511932849.494:51): avc: denied { getattr } for pid=2647 comm=\"sh\" path=\"/usr/bin/systemd-notify\" dev=\"xvda2\" ino=12727806 scontext=system_u:system_r:rabbitmq_t:s0 tcontext=system_u:object_r:systemd_notify_exec_t:s0 tclass=file\ntype=AVC msg=audit(1511932849.494:52): avc: denied { getattr } for pid=2647 comm=\"sh\" path=\"/usr/bin/systemd-notify\" dev=\"xvda2\" ino=12727806 scontext=system_u:system_r:rabbitmq_t:s0 tcontext=system_u:object_r:systemd_notify_exec_t:s0 tclass=file\ntype=LOGIN msg=audit(1511933148.573:71): pid=2719 uid=0 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 old-auid=4294967295 auid=1000 old-ses=4294967295 ses=1 res=1\ntype=USER_CMD msg=audit(1511933154.432:82): pid=2746 uid=1000 auid=1000 ses=1

subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='cwd=\"/home/ec2-user\" cmd=7375202D terminal=pts/0 res=success'\ntype=LOGIN msg=audit(1511935261.397:109): pid=3370 uid=0 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 old-auid=4294967295 auid=0 old-ses=4294967295 ses=2 res=1\ntype=LOGIN msg=audit(1511935477.782:125): pid=3428 uid=0 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 old-auid=4294967295 auid=1000 old-ses=4294967295 ses=3 res=1\ntype=USER_CMD msg=audit(1511935481.780:136): pid=3455 uid=1000 auid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='cwd=\"/home/ec2-user\" cmd=7375202D terminal=pts/1 res=success'\ntype=LOGIN msg=audit(1511938861.466:180): pid=4646 uid=0 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 old-auid=4294967295 auid=0 old-ses=4294967295 ses=4 res=1\ntype=LOGIN msg=audit(1511942461.518:199): pid=5811 uid=0 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 old-auid=4294967295 auid=0 old-ses=4294967295 ses=5 res=1\ntype=LOGIN msg=audit(1511944899.940:269): pid=8632 uid=0 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 old-auid=4294967295 auid=1000 old-ses=4294967295 ses=6 res=1\ntype=USER_CMD msg=audit(1511944904.464:280): pid=8662 uid=1000 auid=1000 ses=6 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='cwd=\"/home/ec2-user\" cmd=7375202D terminal=pts/2 res=success'\ntype=LOGIN msg=audit(1511945100.243:300): pid=8822 uid=0 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 old-auid=4294967295 auid=1000 old-ses=4294967295 ses=7 res=1\ntype=USER_CMD msg=audit(1511945104.249:311): pid=8851 uid=1000 auid=1000 ses=7 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='cwd=\"/home/ec2-user\" cmd=7375202D terminal=pts/3 res=success'\ntype=LOGIN msg=audit(1511946061.575:322): pid=9092 uid=0 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 old-auid=4294967295 auid=0 old-ses=4294967295 ses=8 res=1\ntype=LOGIN msg=audit(1511949661.628:365): pid=9990 uid=0 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 old-auid=4294967295 auid=0 old-ses=4294967295 ses=9 res=1\ntype=LOGIN msg=audit(1511951220.375:391): pid=10493 uid=0 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 old-auid=4294967295 auid=1000 old-ses=4294967295 ses=10 res=1\n\n# LogStash config file for RabbitMQ\n# From: https://confluence.oceanobservatories.org/display/CIDev/Logging+--+logstash+configuration\n\ninput {\n  file {\n    type => \"rabbit\"\n    path => \"/var/log/rabbitmq/*.log\"\n  }\n}\n\nfilter {\n  multiline {\n    pattern => \"^=\"\n    negate => true\n    what => \"previous\"\n  }\n  grok {\n    patterns_dir => \"patterns\"\n    pattern => \"^=%{WORD:report_type} REPORT=+ %{RABBIT_TIME:time_text} ===.*$\"\n  }\n  date {\n    match => [ \"timestamp\" \"dd-MMM-yyyy::HH:mm:ss\" ]\n  }\n  mutate {\n    add_field => [ \"message\", \"%{@message}\" ]\n  }\n  mutate {\n    gsub => [\n      \"message\", \"^=[A-Za-z0-9: =-]+=\\n\", \"\",\n      # interpret message header text as \"severity\"\n      \"report_type\", \"INFO\", \"1\",\n      \"report_type\", \"WARNING\", \"3\",\n      \"report_type\", \"ERROR\", \"4\",\n      \"report_type\", \"CRASH\", \"5\",\n      \"report_type\", \"SUPERVISOR\", \"5\"\n    ]\n  }\n}\n\noutput {\n    rabbitmq {\n        exchange => \"elasticsearch-exchange\"\n        exchange_type => \"direct\"\n        key => \"logstash-routing_key\"\n        host => \"ip-172-31-22-157.eu-west-1.compute.internal\"\n        vhost => es_vhost\n        durable => true\n        persistent => true\n        port => 5672\n        user => \"admin\"\n        password => \"mqadmin\"  \n}\n#  file {\n#    type => \"rabbit\"\n#    path => \"/var/log/lostash-test.out\"\n#    flush_interval => 0\n#  }\n}\n\n", :reason=>"Expected one of #, input, filter, output at line 32, column 1 (byte 874) after ", :level=>:error}

Are you storing a log file in the same directory as your configuration file? Don't do that.

No the logs are in the /var/log directory as mentioned in the conf file. And the conf file resides in /etc/logstash/conf.d

And what files do you have in /etc/logstash/conf.d?

One config each for Rabbit MQ logs, Syslog and Audit logs

What does grep auid=1000 /etc/logstash/conf.d/* return?

Oh sorry I forgot, I had an out file, that I created for getting the errors. I have deleted that now.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.