# Logstash filter plugins not operating on field **URL:** \ **Category:** Logstash\ **Created:** [June 26, 2020, 2:05pm UTC](https://discuss.elastic.co/t/logstash-filter-plugins-not-operating-on-field/238870 "2020-06-26T14:05:28Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Asinus1223](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@Asinus1223](https://discuss.elastic.co/u/Asinus1223)\ **Post date:** [June 26, 2020, 2:05pm UTC](https://discuss.elastic.co/t/logstash-filter-plugins-not-operating-on-field/238870/1 "2020-06-26T14:05:28Z")
I have been working with the notoriously difficult modsecurity\_audit.log in order to gather some useful statistics from it in order to tweak our Web Application firewall. So far, I've been able to separate the different sections A, B, E, F, J, and H (among others), and load them into different filter plugins using the material from here: [https://github.com/bitsofinfo/logstash-modsecurity](https://github.com/bitsofinfo/logstash-modsecurity). This was very helpful, but as the writer said, it really served as a foundation for what I wanted to do. Once I had Section H separated out from the rest of the log, I tried performing some operations on just that section. I cut and pasted just the section H material into a separate log and ran it through its own pipeline so that I wouldn't be distracted by the other material: section.h: ```auto Message: Warning. Pattern match "(?i)[\\s\\S](?:x(?:link:href|html|mlns)|!ENTITY.*?SYSTEM|data:text\\/html|pattern(?=.*?=)|formaction|\\@import|base64)\\b" at ARGS:acceptHeader. [file "/usr/share/modsecurity-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf"] [line "158"] [id "941130"] [rev "2"] [msg "XSS Filter - Category 3: Attribute Vector"] [data "Matched Data: /xhtml found within ARGS:acceptHeader: text/html,application/xhtml xml,application/xml;q=0.9,*/*;q=0.8"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "1"] [accuracy "8"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-xss"] [tag "OWASP_CRS/WEB_ATTACK/XSS"] [tag "WASCTC/WASC-8"] [tag "WASCTC/WASC-22"] [tag "OWASP_TOP_10/A3"] [tag "OWASP_AppSensor/IE1"] [tag "CAPEC-242"] Message: Warning. Pattern match "(?i)[\\s\\S](?:x(?:link:href|html|mlns)|!ENTITY.*?SYSTEM|data:text\\/html|pattern(?=.*?=)|formaction|\\@import|base64)\\b" at ARGS:acceptHeader. [file "/usr/share/modsecurity-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf"] [line "158"] [id "941130"] [rev "2"] [msg "XSS Filter - Category 3: Attribute Vector"] [data "Matched Data: /xhtml found within ARGS:acceptHeader: text/html,application/xhtml xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "1"] [accuracy "8"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-xss"] [tag "OWASP_CRS/WEB_ATTACK/XSS"] [tag "WASCTC/WASC-8"] [tag "WASCTC/WASC-22"] [tag "OWASP_TOP_10/A3"] [tag "OWASP_AppSensor/IE1"] [tag "CAPEC-242"] ``` using the 'dissect' plugin I was able to separate this into three parts: ```auto input { file { path => "/var/log/sectionh.log" type => "section_h" } } filter { dissect { mapping => { "message" => "%{?idea}. %{?before}. %{after}" } output { # turn this off when ready to run in a # real prod environment and get rid of the # "-v" flag when starting logstash stdout { codec => rubydebug } # elasticsearch { # index => "modsecurity-%{+YYYY.MM.dd}" # hosts => ["http://0.0.0.0:9200"] # } #stdout {} ``` Simple, right? Here is the output: ```auto "host" => "elk-stack", "after" => "[file \"/usr/share/modsecurity-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf\"] [line \"158\"] [id \"941130\"] [rev \"2\"] [msg \"XSS Filter - Category 3: Attribute Vector\"] [data \"Matched Data: /xhtml found within ARGS:acceptHeader: text/html,application/xhtml xml,application/xml;q=0.9,*/*;q=0.8\"] [severity \"CRITICAL\"] [ver \"OWASP_CRS/3.0.0\"] [maturity \"1\"] [accuracy \"8\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-xss\"] [tag \"OWASP_CRS/WEB_ATTACK/XSS\"] [tag \"WASCTC/WASC-8\"] [tag \"WASCTC/WASC-22\"] [tag \"OWASP_TOP_10/A3\"] [tag \"OWASP_AppSensor/IE1\"] [tag \"CAPEC-242\"]", "message" => "Message: Warning. Pattern match \"(?i)[\\\\s\\\\S](?:x(?:link:href|html|mlns)|!ENTITY.*?SYSTEM|data:text\\\\/html|pattern(?=.*?=)|formaction|\\\\@import|base64)\\\\b\" at ARGS:acceptHeader. [file \"/usr/share/modsecurity-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf\"] [line \"158\"] [id \"941130\"] [rev \"2\"] [msg \"XSS Filter - Category 3: Attribute Vector\"] [data \"Matched Data: /xhtml found within ARGS:acceptHeader: text/html,application/xhtml xml,application/xml;q=0.9,*/*;q=0.8\"] [severity \"CRITICAL\"] [ver \"OWASP_CRS/3.0.0\"] [maturity \"1\"] [accuracy \"8\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-xss\"] [tag \"OWASP_CRS/WEB_ATTACK/XSS\"] [tag \"WASCTC/WASC-8\"] [tag \"WASCTC/WASC-22\"] [tag \"OWASP_TOP_10/A3\"] [tag \"OWASP_AppSensor/IE1\"] [tag \"CAPEC-242\"]", "type" => "section_h", "@timestamp" => 2020-06-26T13:41:33.496Z, "path" => "/var/log/sectionh.log", "@version" => "1" } ``` As you can see, the "after" field has the information I'm interested in. This is what id like to have from the "after" field to store in Elasticsearch: ```auto "file" => "/usr/share/modsecurity-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf" "line" => 158 "id" => "941130 "rev" => "2" "msg" => "XSS Filter - Category 3: Attribute Vector" "data" => "Matched Data: /xhtml found within ARGS:acceptHeader: text/html,application/xhtml xml,application/xml;q=0.9,*/*;q=0.8" "severity" => "CRITICAL" "ver" => "OWASP_CRS/3.0.0\" "maturity" => "1" "accuracy" => "8" "tag" => "application-multi" "tag" => "language-multi" "tag" => "platform-multi" "tag" => "attack-xss" "tag" =>"OWASP_CRS/WEB_ATTACK/XSS" "tag" => "WASCTC/WASC-8" "tag" => "WASCTC/WASC-22" "tag" => "OWASP_TOP_10/A3" "tag" => "OWASP_AppSensor/IE1" [ "tag" => "CAPEC-242" ``` It doesn't look as though it would be that hard, does it? Yet for some reason, any filter plugin I try to apply to the "after" field doesn't 'take', and the output doesn't change. I have tried both the 'kv' and the 'gsub' plugins, and neither of them modify the "after" fields as my interpretation of the documentation led me to believe that they would. Here is the full configuration file for the pipeline: ```auto input { file { path => "/var/log/sectionh.log" type => "section_h" } } filter { dissect { mapping => { "message" => "%{?idea}. %{?before}. %{after}" } } mutate { remove_field => ["message"] gsub => [ "after", "\[\"", ""] } } filter { kv { source => "after" allow_duplicate_values => true include_brackets => false } } output { # turn this off when ready to run in a # real prod environment and get rid of the # "-v" flag when starting logstash stdout { codec => rubydebug } # elasticsearch { # index => "modsecurity-%{+YYYY.MM.dd}" # hosts => ["http://0.0.0.0:9200"] # } #stdout {} } Despite everything, the pluins do not operate on the "after" field. They don't appear to be operating on anything. Is there some problem with the nesting or the placement? I have been working on this for a week. Thanking you all in anticipation: ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 26, 2020, 3:08pm UTC](https://discuss.elastic.co/t/logstash-filter-plugins-not-operating-on-field/238870/2 "2020-06-26T15:08:07Z")
I do not think the mutate+gsub has any effect. The pattern does not match. You can parse the [after] field using ``` kv { source => "after" value_split => " " field_split_pattern => "\] \[|^\[|\]$" allow_duplicate_values => true } ``` which produces ``` "tag" => [ [0] "application-multi", [1] "language-multi", [2] "platform-multi", [3] "attack-xss", [4] "OWASP_CRS/WEB_ATTACK/XSS", [5] "WASCTC/WASC-8", [6] "WASCTC/WASC-22", [7] "OWASP_TOP_10/A3", [8] "OWASP_AppSensor/IE1", [9] "CAPEC-242" ], "file" => "/usr/share/modsecurity-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf", "id" => "941130", "rev" => "2", "line" => "158", "msg" => "XSS Filter - Category 3: Attribute Vector", "severity" => "CRITICAL", "ver" => "OWASP_CRS/3.0.0", "maturity" => "1", "data" => "Matched Data: /xhtml found within ARGS:acceptHeader: text/html,application/xhtml xml,application/xml;q=0.9,*/*;q=0.8", "accuracy" => "8", ``` ---
**Author:** ![Asinus1223](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@Asinus1223](https://discuss.elastic.co/u/Asinus1223)\ **Post date:** [June 29, 2020, 12:54pm UTC](https://discuss.elastic.co/t/logstash-filter-plugins-not-operating-on-field/238870/3 "2020-06-29T12:54:42Z")
Thank you, Badger. That worked perfectly. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 27, 2020, 12:54pm UTC](https://discuss.elastic.co/t/logstash-filter-plugins-not-operating-on-field/238870/4 "2020-07-27T12:54:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.