Removing special characters

Elastic Stack Logstash
1

Hello,

I have some special characters in one of my fields which I'm having issues removing. They may be reserved characters so my logstash instance is failing. Here is the event in question...

"proof" => [
            [0] "\"Vulnerable software installed: Apache Log4j Core 2.11.1 (/var/lib/docker/overlay2/0edef2ae5c6fc053ca616bd84b6bfe81e39ebdc071ca44c9c29c378112693860/diff/usr/share/elasticsearch/lib/log4j-core-2.11.1.jar)No mitigation applied. JndiLookup class discovered.\"",
            [1] "\"Vulnerable software installed: Apache Log4j Core 2.14.0 (/var/lib/docker/overlay2/c2877f2cdbd6d80bcafb50907a452a74eefa6e95a34de050908632c0079c0b3a/diff/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.14.0.jar)No mitigation applied. JndiLookup class discovered.\"",
            [2] "\"Vulnerable software installed: Apache Log4j Core 2.11.1 (/var/lib/docker/overlay2/9e668265b4cc434f8aa963a773a06c0662323ec76bdd91bf012b4db643298d3a/diff/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.11.1.jar)No mitigation applied. JndiLookup class discovered.\"",
            [3] "\"Vulnerable software installed: Apache Log4j Core 2.11.1 (/var/lib/docker/overlay2/04b58f14d8ef5bab515849cb8ac65fd4843915ef905a3fadd97268a5dfd8b79a/diff/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.11.1.jar)No mitigation applied. JndiLookup class discovered.\"",
            [4] "\"Vulnerable software installed: Apache Log4j Core 2.14.0 (/var/lib/docker/overlay2/b2efd746948b7132e32c25d9812d7ed46d27ca6ed4a98835386628d838c16e5b/diff/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.14.0.jar)No mitigation applied. JndiLookup class discovered.\"",
            [5] "\"Vulnerable software installed: Apache Log4j Core 2.11.1 (/var/lib/docker/overlay2/3fca162b169ea32445c5a3e1f554891025dc1c24f9f9bf30ddc4af95a0670194/diff/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.11.1.jar)No mitigation applied. JndiLookup class discovered.\""
        ],

My config (currently)

  mutate {
    gsub => [
      "proof", "[{}]", "",
    split => { "proof" => "," }
  }
  mutate { strip => [ "proof" ] }

In my output I have extra " and \ from the original data which was one message. I'm attempting to remove the special characters and performing a split on the event to create an array. So far the array works fine, I'm just left with additional characters so I need to clean up the data. Every time I attempt to add those characters to the gsub I get an error exception=>#<RegexpError: premature end of char-class: /[\]/>

Any assistance is appreciated!

Thank you.

2

What characters do you want to remove?

3

Hi Badger

The backslashes and extra double quotes.

4

There are no backslashes in your message. The rubydebug format inserts a backslash to escape the double quote.

split { field => "proof" }
mutate { gsub => [ "proof", '"', "" ] }
5

I think there is a backslash. I gave you the Ruby Debug output but here is the JSON data in the Elastic document when I do a _doc query.

"proof" : [
        "\"Vulnerable software installed: Apache Log4j Core 2.11.1 (/var/lib/docker/overlay2/0edef2ae5c6fc053ca616bd84b6bfe81e39ebdc071ca44c9c29c378112693860/diff/usr/share/elasticsearch/lib/log4j-core-2.11.1.jar)No mitigation applied. JndiLookup class discovered.\"",
        "\"Vulnerable software installed: Apache Log4j Core 2.14.0 (/var/lib/docker/overlay2/c2877f2cdbd6d80bcafb50907a452a74eefa6e95a34de050908632c0079c0b3a/diff/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.14.0.jar)No mitigation applied. JndiLookup class discovered.\"",
        "\"Vulnerable software installed: Apache Log4j Core 2.11.1 (/var/lib/docker/overlay2/9e668265b4cc434f8aa963a773a06c0662323ec76bdd91bf012b4db643298d3a/diff/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.11.1.jar)No mitigation applied. JndiLookup class discovered.\"",
        "\"Vulnerable software installed: Apache Log4j Core 2.11.1 (/var/lib/docker/overlay2/04b58f14d8ef5bab515849cb8ac65fd4843915ef905a3fadd97268a5dfd8b79a/diff/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.11.1.jar)No mitigation applied. JndiLookup class discovered.\"",
        "\"Vulnerable software installed: Apache Log4j Core 2.14.0 (/var/lib/docker/overlay2/b2efd746948b7132e32c25d9812d7ed46d27ca6ed4a98835386628d838c16e5b/diff/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.14.0.jar)No mitigation applied. JndiLookup class discovered.\"",
        "\"Vulnerable software installed: Apache Log4j Core 2.11.1 (/var/lib/docker/overlay2/3fca162b169ea32445c5a3e1f554891025dc1c24f9f9bf30ddc4af95a0670194/diff/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.11.1.jar)No mitigation applied. JndiLookup class discovered.\""
      ]
6

I still do not think there is a backslash in your field, but you can try

mutate { gsub => [ "proof", '"', "", "proof", "[\\]", "" ] }
7

That did the trick. Thank you Badger.

"proof" : [
        "Vulnerable software installed: Apache Log4j Core 2.11.1 (/var/lib/docker/overlay2/0edef2ae5c6fc053ca616bd84b6bfe81e39ebdc071ca44c9c29c378112693860/diff/usr/share/elasticsearch/lib/log4j-core-2.11.1.jar)No mitigation applied. JndiLookup class discovered.",
        "Vulnerable software installed: Apache Log4j Core 2.14.0 (/var/lib/docker/overlay2/b2efd746948b7132e32c25d9812d7ed46d27ca6ed4a98835386628d838c16e5b/diff/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.14.0.jar)No mitigation applied. JndiLookup class discovered.",
        "Vulnerable software installed: Apache Log4j Core 2.11.1 (/var/lib/docker/overlay2/04b58f14d8ef5bab515849cb8ac65fd4843915ef905a3fadd97268a5dfd8b79a/diff/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.11.1.jar)No mitigation applied. JndiLookup class discovered.",
        "Vulnerable software installed: Apache Log4j Core 2.11.1 (/var/lib/docker/overlay2/9e668265b4cc434f8aa963a773a06c0662323ec76bdd91bf012b4db643298d3a/diff/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.11.1.jar)No mitigation applied. JndiLookup class discovered.",
        "Vulnerable software installed: Apache Log4j Core 2.11.1 (/var/lib/docker/overlay2/3fca162b169ea32445c5a3e1f554891025dc1c24f9f9bf30ddc4af95a0670194/diff/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.11.1.jar)No mitigation applied. JndiLookup class discovered.",
        "Vulnerable software installed: Apache Log4j Core 2.14.0 (/var/lib/docker/overlay2/c2877f2cdbd6d80bcafb50907a452a74eefa6e95a34de050908632c0079c0b3a/diff/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.14.0.jar)No mitigation applied. JndiLookup class discovered."
      ],


  mutate {
    gsub => [
      "proof", '"', "", "proof", "[\\]", "", "proof", "[{}]", "" ]
    split => { "proof" => "," }
  }
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.