Filebeat to parse modsecurity json logs

[angeL]

Hi team,

I'm using filebeat 6.2.4 to parse ModSecurity logs in Json format, so far so good using the following config:

    filebeat.prospectors:
    - type: log
      paths:
        - /var/log/httpd/modsec_audit.log
      json.keys_under_root: true
      json.add_error_key: true
    output.elasticsearch:
      hosts: ["192.168.1.128:9200"]

This is my sample log:

{"body":["j_username=%3Cscript%3Ealert%28%27xss%27%29%3C%2Fscript%3E&j_password=*******&next_url=%2Faction%2Fmy-portal"]},"response":{"protocol":"HTTP/1.1","status":403,"headers":{"X-Frame-Options":"SAMEORIGIN","Strict-Transport-Security":"max-age=3600","X-Xss-Protection":"1; mode=block","X-Content-Type-Options":"nosniff","X-Permitted-Cross-Domain-Policies":"none","Last-Modified":"Wed, 04 Apr 2018 18:40:51 GMT","ETag":"\"47d-5690a298d6a06\"","Accept-Ranges":"bytes","Content-Length":"1149","Keep-Alive":"timeout=5, max=100","Connection":"Keep-Alive","Content-Type":"text/html"}},"audit_data":{"messages":["Warning. detected XSS using libinjection. [file \"/etc/httpd/modsecurity/owasp-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf\"] [line \"64\"] [id \"941100\"] [rev \"2\"] [msg \"XSS Attack Detected via libinjection\"] [data \"Matched Data: upgrade-insecure-requests found within ARGS:j_username: <script>alert('xss')</script>\"] [severity \"CRITICAL\"] [ver \"OWASP_CRS/3.0.0\"] [maturity \"1\"] [accuracy \"9\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-xss\"] [tag \"OWASP_CRS/WEB_ATTACK/XSS\"] [tag \"WASCTC/WASC-8\"] [tag \"WASCTC/WASC-22\"] [tag \"OWASP_TOP_10/A3\"] [tag \"OWASP_AppSensor/IE1\"] [tag \"CAPEC-242\"]","Warning. Pattern match \"(?i)([<\\xef\\xbc\\x9c]script[^>\\xef\\xbc\\x9e]*[>\\xef\\xbc\\x9e][\\\\s\\\\S]*?)\" at ARGS:j_username. [file \"/etc/httpd/modsecurity/owasp-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf\"] [line \"99\"] [id \"941110\"] [rev \"2\"] [msg \"XSS Filter - Category 1: Script Tag Vector\"] [data \"Matched Data: <script> found within ARGS:j_username: <script>alert('xss')</script>\"] [severity \"CRITICAL\"] [ver \"OWASP_CRS/3.0.0\"] [maturity \"4\"] [accuracy \"9\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-xss\"] [tag \"OWASP_CRS/WEB_ATTACK/XSS\"] [tag \"WASCTC/WASC-8\"] [tag \"WASCTC/WASC-22\"] [tag \"OWASP_TOP_10/A3\"] [tag \"OWASP_AppSensor/IE1\"] [tag \"CAPEC-242\"]","Warning. Pattern match \"(?i)<[^\\\\w<>]*(?:[^<>\\\"'\\\\s]*:)?[^\\\\w<>]*(?:\\\\W*?s\\\\W*?c\\\\W*?r\\\\W*?i\\\\W*?p\\\\W*?t|\\\\W*?f\\\\W*?o\\\\W*?r\\\\W*?m|\\\\W*?s\\\\W*?t\\\\W*?y\\\\W*?l\\\\W*?e|\\\\W*?s\\\\W*?v\\\\W*?g|\\\\W*?m\\\\W*?a\\\\W*?r\\\\W*?q\\\\W*?u\\\\W*?e\\\\W*?e|(?:\\\\W*?l\\\\W*?i\\\\W*?n\\\\W*?k|\\\\W*?o\\\\W*?b\\\\W*?j\\\\W*?e\\ ...\" at ARGS:j_username. [file \"/etc/httpd/modsecurity/owasp-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf\"] [line \"236\"] [id \"941160\"] [rev \"2\"] [msg \"NoScript XSS InjectionChecker: HTML Injection\"] [data \"Matched Data: <script found within ARGS:j_username: <script>alert('xss')</script>\"] [severity \"CRITICAL\"] [ver \"OWASP_CRS/3.0.0\"] [maturity \"1\"] [accuracy \"8\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-xss\"] [tag \"OWASP_CRS/WEB_ATTACK/XSS\"] [tag \"WASCTC/WASC-8\"] [tag \"WASCTC/WASC-22\"] [tag \"OWASP_TOP_10/A3\"] [tag \"OWASP_AppSensor/IE1\"] [tag \"CAPEC-242\"]","Access denied with code 403 (phase 2). Operator GE matched 5 at TX:anomaly_score. [file \"/etc/httpd/modsecurity/owasp-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf\"] [line \"57\"] [id \"949110\"] [msg \"Inbound Anomaly Score Exceeded (Total Score: 15)\"] [severity \"CRITICAL\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-generic\"]","Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file \"/etc/httpd/modsecurity/owasp-crs/rules/RESPONSE-980-CORRELATION.conf\"] [line \"73\"] [id \"980130\"] [msg \"Inbound Anomaly Score Exceeded (Total Inbound Score: 15 - SQLI=0,XSS=15,RFI=0,LFI=0,RCE=0,PHPI=0,HTTP=0,SESS=0): NoScript XSS InjectionChecker: HTML Injection\"] [tag \"event-correlation\"]"],"

When filebeat gets audit_data.messages he gets the entire message as a single field, is there any way I can get sub fields in the audit_data.messages like id or msg so I can filter in Kibana with thoose fields?

Hope you can help me!

[jsoriano (Jaime Soriano)]

Hi @angeL and welcome to discuss

I wonder how you are obtaining these JSON logs for modsecurity, could you share its logging configuration? With this structure it looks difficult to parse.

[angeL]

Hi jsoriano, thanks for the welcome!

Since Version 2.9.1 Mod Security accepts json format as log structure, this is my config:

# Log everything we know about a transaction.
SecAuditLogParts ABCIJDEFHZ
SecAuditLogFormat JSON

# Use a single file for logging. This is much easier to look at, but
# assumes that you will use the audit log only ocassionally.
#
SecAuditLogType Serial
SecAuditLog /var/log/httpd/modsec_audit.log

SecAuditLogParts defines which information you want to log, in this guide you can see the meaning of each letter: https://github.com/SpiderLabs/ModSecurity/wiki/ModSecurity-2-Data-Formats but in general words:

A: Http Audit Log header
B: Http Request headers
C: Request Body
H: Audit Log Trailer (contains sub fields like message, server)
Z: End of log

I see that filebeat is parsing well every field from the json log, but the message field from the "H" Audit Log Trailer has sub fields separated by [], that info is really useful for troubleshooting the WAF and creating visualizations like top matching rules, attacks, etc.

[jsoriano (Jaime Soriano)]

I'm afraid that we don't have anything out-of-the-box for this on Beats, but you can try to define your own ingest pipeline

You'd need to use the foreach processor on the audit_data.messages field, and the grok or script processors to parse each one of the messages.

If you decide to try this we'd be happy to guide you to contribute this as a new filebeat module. Filebeat modules contain pipelines, field mappings and/or dashboards that are useful for an specific application, I think it could be a good idea to have a module for modsecurity. If you cannot do it, feel free to open a new issue requesting this new module.

[angeL]

Hi jsoriano, I'll give it a try to grok.

How can I filter a number by size?

I want to get the Id field which is always a 6 number digit.

Thanks for the advice

[jsoriano (Jaime Soriano)]

Take a look to the documentation for elastic ingest processor where you can find useful information about writing grok patterns as well as some examples and the patterns included by default.

[angeL]

Ok this is the config I've done so far:

Filebeat:

filebeat.prospectors:
- type: log
  paths:
    - /var/log/httpd/modsec_audit.log
  json.keys_under_root: true
  json.add_error_key: true
output.logstash:
  hosts: ["192.168.1.128:5044"]

Now I'm using logstash to recieve the logs and filter with grok:

input {
    beats {
        port => "5044"
    }
}
filter {
  grok {
    patterns_dir => "/etc/logstash/conf.d/patterns"
    match => { "messages" => "%{UNIXPATH:FileRule}" }
    add_field => { "File_Rule" => "%{FileRule}" }
    tag_on_failure => ["grok_fail"]
  }
}
output {
  file {
  path => "/tmp/filebeat"
  }
}

When I check the file I see the field was not created and the tag grok_fail appears in the log file, any suggestions?

I also tested the filter on the http://grokdebug.herokuapp.com/ and it's working fine.

[angeL]

Any suggestions about how can I makes this work?

[jsoriano (Jaime Soriano)]

Hi @angeL,

Sorry for my late reply.

You shouldn't need logstash to do it, you can do it with a filebeat module that includes an ingest pipeline. Did you try to follow the instructions to create a new module?

In any case your logstash configuration looks fine, the only thing I see is that the field you are parsing is not messages but audit_data.messages, isn't it? Could you try to change this?