# Filebeat to parse modsecurity json logs

**URL:** <https://discuss.elastic.co/t/filebeat-to-parse-modsecurity-json-logs/134662>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 5, 2018, 3:56pm UTC](https://discuss.elastic.co/t/filebeat-to-parse-modsecurity-json-logs/134662 "2018-06-05T15:56:10Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![angeL](https://avatars.discourse-cdn.com/v4/letter/a/49beb7/32.png) [@angeL](https://discuss.elastic.co/u/angeL)\
**Post date:** [June 5, 2018, 3:56pm UTC](https://discuss.elastic.co/t/filebeat-to-parse-modsecurity-json-logs/134662/1 "2018-06-05T15:56:10Z")

</div>

Hi team,

I'm using filebeat 6.2.4 to parse ModSecurity logs in Json format, so far so good using the following config:

```
    filebeat.prospectors:
    - type: log
      paths:
        - /var/log/httpd/modsec_audit.log
      json.keys_under_root: true
      json.add_error_key: true
    output.elasticsearch:
      hosts: ["192.168.1.128:9200"]

```

This is my sample log:

`{"body":["j_username=%3Cscript%3Ealert%28%27xss%27%29%3C%2Fscript%3E&j_password= ******* &next_url=%2Faction%2Fmy-portal"]},"response":{"protocol":"HTTP/1.1","status":403,"headers":{"X-Frame-Options":"SAMEORIGIN","Strict-Transport-Security":"max-age=3600","X-Xss-Protection":"1; mode=block","X-Content-Type-Options":"nosniff","X-Permitted-Cross-Domain-Policies":"none","Last-Modified":"Wed, 04 Apr 2018 18:40:51 GMT","ETag":"\"47d-5690a298d6a06\"","Accept-Ranges":"bytes","Content-Length":"1149","Keep-Alive":"timeout=5, max=100","Connection":"Keep-Alive","Content-Type":"text/html"}},"audit_data":{"messages":["Warning. detected XSS using libinjection. [file \"/etc/httpd/modsecurity/owasp-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf\"] [line \"64\"] [id \"941100\"] [rev \"2\"] [msg \"XSS Attack Detected via libinjection\"] [data \"Matched Data: upgrade-insecure-requests found within ARGS:j_username: <script>alert('xss')</script>\"] [severity \"CRITICAL\"] [ver \"OWASP_CRS/3.0.0\"] [maturity \"1\"] [accuracy \"9\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-xss\"] [tag \"OWASP_CRS/WEB_ATTACK/XSS\"] [tag \"WASCTC/WASC-8\"] [tag \"WASCTC/WASC-22\"] [tag \"OWASP_TOP_10/A3\"] [tag \"OWASP_AppSensor/IE1\"] [tag \"CAPEC-242\"]","Warning. Pattern match \"(?i)([<\\xef\\xbc\\x9c]script[^>\\xef\\xbc\\x9e]*[>\\xef\\xbc\\x9e][\\\\s\\\\S]*?)\" at ARGS:j_username. [file \"/etc/httpd/modsecurity/owasp-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf\"] [line \"99\"] [id \"941110\"] [rev \"2\"] [msg \"XSS Filter - Category 1: Script Tag Vector\"] [data \"Matched Data: <script> found within ARGS:j_username: <script>alert('xss')</script>\"] [severity \"CRITICAL\"] [ver \"OWASP_CRS/3.0.0\"] [maturity \"4\"] [accuracy \"9\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-xss\"] [tag \"OWASP_CRS/WEB_ATTACK/XSS\"] [tag \"WASCTC/WASC-8\"] [tag \"WASCTC/WASC-22\"] [tag \"OWASP_TOP_10/A3\"] [tag \"OWASP_AppSensor/IE1\"] [tag \"CAPEC-242\"]","Warning. Pattern match \"(?i)<[^\\\\w<>]*(?:[^<>\\\"'\\\\s]*:)?[^\\\\w<>]*(?:\\\\W*?s\\\\W*?c\\\\W*?r\\\\W*?i\\\\W*?p\\\\W*?t|\\\\W*?f\\\\W*?o\\\\W*?r\\\\W*?m|\\\\W*?s\\\\W*?t\\\\W*?y\\\\W*?l\\\\W*?e|\\\\W*?s\\\\W*?v\\\\W*?g|\\\\W*?m\\\\W*?a\\\\W*?r\\\\W*?q\\\\W*?u\\\\W*?e\\\\W*?e|(?:\\\\W*?l\\\\W*?i\\\\W*?n\\\\W*?k|\\\\W*?o\\\\W*?b\\\\W*?j\\\\W*?e\\ ...\" at ARGS:j_username. [file \"/etc/httpd/modsecurity/owasp-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf\"] [line \"236\"] [id \"941160\"] [rev \"2\"] [msg \"NoScript XSS InjectionChecker: HTML Injection\"] [data \"Matched Data: <script found within ARGS:j_username: <script>alert('xss')</script>\"] [severity \"CRITICAL\"] [ver \"OWASP_CRS/3.0.0\"] [maturity \"1\"] [accuracy \"8\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-xss\"] [tag \"OWASP_CRS/WEB_ATTACK/XSS\"] [tag \"WASCTC/WASC-8\"] [tag \"WASCTC/WASC-22\"] [tag \"OWASP_TOP_10/A3\"] [tag \"OWASP_AppSensor/IE1\"] [tag \"CAPEC-242\"]","Access denied with code 403 (phase 2). Operator GE matched 5 at TX:anomaly_score. [file \"/etc/httpd/modsecurity/owasp-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf\"] [line \"57\"] [id \"949110\"] [msg \"Inbound Anomaly Score Exceeded (Total Score: 15)\"] [severity \"CRITICAL\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-generic\"]","Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file \"/etc/httpd/modsecurity/owasp-crs/rules/RESPONSE-980-CORRELATION.conf\"] [line \"73\"] [id \"980130\"] [msg \"Inbound Anomaly Score Exceeded (Total Inbound Score: 15 - SQLI=0,XSS=15,RFI=0,LFI=0,RCE=0,PHPI=0,HTTP=0,SESS=0): NoScript XSS InjectionChecker: HTML Injection\"] [tag \"event-correlation\"]"],"`

When filebeat gets audit\_data.messages he gets the entire message as a single field, is there any way I can get sub fields in the audit\_data.messages like id or msg so I can filter in Kibana with thoose fields?

Hope you can help me!

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [June 5, 2018, 4:29pm UTC](https://discuss.elastic.co/t/filebeat-to-parse-modsecurity-json-logs/134662/2 "2018-06-05T16:29:39Z")

</div>

Hi @angeL and welcome to discuss 🙂

I wonder how you are obtaining these JSON logs for modsecurity, could you share its logging configuration? With this structure it looks difficult to parse.

---

<div class="post-metadata">

**Author:** ![angeL](https://avatars.discourse-cdn.com/v4/letter/a/49beb7/32.png) [@angeL](https://discuss.elastic.co/u/angeL)\
**Post date:** [June 5, 2018, 5:38pm UTC](https://discuss.elastic.co/t/filebeat-to-parse-modsecurity-json-logs/134662/3 "2018-06-05T17:38:38Z")

</div>

Hi jsoriano, thanks for the welcome!

Since Version 2.9.1 Mod Security accepts json format as log structure, this is my config:

```
# Log everything we know about a transaction.
SecAuditLogParts ABCIJDEFHZ
SecAuditLogFormat JSON

# Use a single file for logging. This is much easier to look at, but
# assumes that you will use the audit log only ocassionally.
#
SecAuditLogType Serial
SecAuditLog /var/log/httpd/modsec_audit.log

```

SecAuditLogParts defines which information you want to log, in this guide you can see the meaning of each letter: [https://github.com/SpiderLabs/ModSecurity/wiki/ModSecurity-2-Data-Formats](https://github.com/SpiderLabs/ModSecurity/wiki/ModSecurity-2-Data-Formats) but in general words:

A: Http Audit Log header  
B: Http Request headers  
C: Request Body  
H: Audit Log Trailer (contains sub fields like message, server)  
Z: End of log

I see that filebeat is parsing well every field from the json log, but the message field from the "H" Audit Log Trailer has sub fields separated by [], that info is really useful for troubleshooting the WAF and creating visualizations like top matching rules, attacks, etc.

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [June 5, 2018, 6:58pm UTC](https://discuss.elastic.co/t/filebeat-to-parse-modsecurity-json-logs/134662/4 "2018-06-05T18:58:16Z")

</div>

I'm afraid that we don't have anything out-of-the-box for this on Beats, but you can try to define your own [ingest pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/master/pipeline.html) 🙂

You'd need to use the [foreach](https://www.elastic.co/guide/en/elasticsearch/reference/master/foreach-processor.html) processor on the `audit_data.messages` field, and the [grok](https://www.elastic.co/guide/en/elasticsearch/reference/master/grok-processor.html) or [script](https://www.elastic.co/guide/en/elasticsearch/reference/master/script-processor.html) processors to parse each one of the messages.

If you decide to try this we'd be happy to guide you to contribute this as a [new filebeat module](https://www.elastic.co/guide/en/beats/devguide/6.2/filebeat-modules-devguide.html). Filebeat [modules](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules.html) contain pipelines, field mappings and/or dashboards that are useful for an specific application, I think it could be a good idea to have a module for modsecurity. If you cannot do it, feel free to [open a new issue](https://github.com/elastic/beats/issues/new) requesting this new module.

---

<div class="post-metadata">

**Author:** ![angeL](https://avatars.discourse-cdn.com/v4/letter/a/49beb7/32.png) [@angeL](https://discuss.elastic.co/u/angeL)\
**Post date:** [June 6, 2018, 3:59pm UTC](https://discuss.elastic.co/t/filebeat-to-parse-modsecurity-json-logs/134662/5 "2018-06-06T15:59:34Z")

</div>

Hi jsoriano, I'll give it a try to grok.

How can I filter a number by size?

I want to get the Id field which is always a 6 number digit.

Thanks for the advice 😀

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [June 6, 2018, 4:42pm UTC](https://discuss.elastic.co/t/filebeat-to-parse-modsecurity-json-logs/134662/6 "2018-06-06T16:42:40Z")

</div>

Take a look to the documentation for elastic ingest [processor](https://www.elastic.co/guide/en/elasticsearch/reference/master/grok-processor.html) where you can find useful information about writing grok patterns as well as some examples and the [patterns included by default](https://github.com/elastic/elasticsearch/tree/master/libs/grok/src/main/resources/patterns).

---

<div class="post-metadata">

**Author:** ![angeL](https://avatars.discourse-cdn.com/v4/letter/a/49beb7/32.png) [@angeL](https://discuss.elastic.co/u/angeL)\
**Post date:** [June 7, 2018, 4:39pm UTC](https://discuss.elastic.co/t/filebeat-to-parse-modsecurity-json-logs/134662/7 "2018-06-07T16:39:38Z")

</div>

Ok this is the config I've done so far:

Filebeat:

```
filebeat.prospectors:
- type: log
  paths:
    - /var/log/httpd/modsec_audit.log
  json.keys_under_root: true
  json.add_error_key: true
output.logstash:
  hosts: ["192.168.1.128:5044"]

```

Now I'm using logstash to recieve the logs and filter with grok:

```
input {
    beats {
        port => "5044"
    }
}
filter {
  grok {
    patterns_dir => "/etc/logstash/conf.d/patterns"
    match => { "messages" => "%{UNIXPATH:FileRule}" }
    add_field => { "File_Rule" => "%{FileRule}" }
    tag_on_failure => ["grok_fail"]
  }
}
output {
  file {
  path => "/tmp/filebeat"
  }
}

```

When I check the file I see the field was not created and the tag grok\_fail appears in the log file, any suggestions?

I also tested the filter on the [http://grokdebug.herokuapp.com/](http://grokdebug.herokuapp.com/) and it's working fine.

---

<div class="post-metadata">

**Author:** ![angeL](https://avatars.discourse-cdn.com/v4/letter/a/49beb7/32.png) [@angeL](https://discuss.elastic.co/u/angeL)\
**Post date:** [June 11, 2018, 1:35pm UTC](https://discuss.elastic.co/t/filebeat-to-parse-modsecurity-json-logs/134662/8 "2018-06-11T13:35:13Z")

</div>

Any suggestions about how can I makes this work?

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [June 21, 2018, 3:24pm UTC](https://discuss.elastic.co/t/filebeat-to-parse-modsecurity-json-logs/134662/9 "2018-06-21T15:24:34Z")

</div>

Hi @angeL,

Sorry for my late reply.

You shouldn't need logstash to do it, you can do it with a filebeat module that includes an ingest pipeline. Did you try to follow the [instructions to create a new module](https://www.elastic.co/guide/en/beats/devguide/6.3/filebeat-modules-devguide.html)?

In any case your logstash configuration looks fine, the only thing I see is that the field you are parsing is not `messages` but `audit_data.messages`, isn't it? Could you try to change this?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 19, 2018, 3:24pm UTC](https://discuss.elastic.co/t/filebeat-to-parse-modsecurity-json-logs/134662/10 "2018-07-19T15:24:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.