Remove wildcard fields in logstash

[Yanick_Quirion]

Hello,

I have this data:

"winlog" => {
    "channel" => "Security",
    "record_id" => 1401770357,
    "event_data" => {
        "param14" => "-",
        "param9" => "-",
        "param21" => "-",
        "param13" => "-",
        "param19" => "-",
        "param15" => "-",
        "param3" => "coded\",\"Accept\":\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\",\"Accept-Encoding\":\"gzip, deflate, br\",\"Accept-Language\":\"fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3\",\"Cookie\":\"_ga=GA1.2.1774089451.1521394333; _fbp=fb.1.1551118745254.676256511; _fbc=fb.1.1592585214629.IwAR0dWXKxMpI_oreMHAmH__1b7QWeoVc-Uwob6Bo_dLQN2ohFNUsC6IgrzuA; _hjid=a468c35d-cc62-463c-9768-f6a0b16ba200; ",
        "param18" => "-",
        "param16" => "-",
        "param12" => "-",
        "param5" => "-",
        "param4" => "_gcl_au=1.1.2121093392.1596402799\",\"Expect\":\"100-continue\",\"Host\":\"authentification.usherbrooke.ca\",\"Referer\":\"https://cas.usherbrooke.ca/login\",\"User-Agent\":\"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:80.0) Gecko/20100101 Firefox/80.0\"}",
        "param17" => "-",
        "param8" => "-",
        "param10" => "-",
        "param6" => "-",
        "param11" => "-",
        "param20" => "-",
        "param2" => "{\"Origin\":\"https://cas.usherbrooke.ca\",\"Upgrade-Insecure-Requests\":\"1\",\"X-Forwarded-For\":\"70.83.116.71\",\"X-MS-Forwarded-Client-IP\":\"70.83.116.71,132.210.7.250\",\"X-MS-ADFS-Proxy-Client-IP\":\"132.210.7.250\",\"client-request-id\":\"2225ca06-d83e-4867-52e8-018000080032\",\"X-MS-Proxy\":\"adfsproxy41\",\"X-MS-Endpoint-Absolute-Path\":\"/adfs/ls/\",\"Content-Length\":\"8304\",\"Content-Type\":\"application/x-www-form-urlen",
        "param7" => "-"
     }

I want to remove all field ending by param1, param2, etc.. (param\d+).

I'm not very good with ruby code. I've found this code on this side:

ruby {
    code => '
        event.get("winlog").each { |k, v|
            if v.is_a? Hash and v.key? "param1"
                event.remove("[winlog][#{k}][param1]")
            end
        }
    '
  }

I can remove "param1" but I would like to remove all field ending param\d+.

The code I found comes from this article: Wildcards in logstash remove_field

Can someone can help me out with this ruby problem?

Thank you all and best regards,
Yanick

[Badger]

Why not just remove the [winlog][event_data] field, which you could do using a mutate filter?

[Yanick_Quirion]

Hi Badger,

That's a good idea, but sometimes there are some pertinent information such as [winlog][event_data][hostname]. So I can't remove this entire branch, just the [winlog][event_data][param\d+].

Thank you for you answer and best regards,
Yanick

[Badger]

OK, so try something like this (which I have not tested)

ruby {
    code => '
        winlog = event.get("winlog")
        if winlog.is_a?(Hash) and winlog.key?("event_data")
            if winlog["event_data"].is_a?(Hash)
                winlog["event_data"].each { k, v }
                if /^param\d+/.match(k)
                    event.remove("[winlog][event_data][#{k}]")
                end
            end
        end
    '
}

[Yanick_Quirion]

Hi Badger!

Thank you so much for your help. After doing a slight change, it worked like a charm!

I would like to be smart as you!!

Here is the final code:

  ruby {
    code => '
      winlog = event.get("winlog")
        if winlog.is_a?(Hash) and winlog.key?("event_data")
          if winlog["event_data"].is_a?(Hash)
            winlog["event_data"].each { |k, v| 
              if /^param\d+/.match(k)
                event.remove("[winlog][event_data][#{k}]")
              end
            }
          end
      end
    '
  }

Thank you so much! If you have a good place to learn ruby coding into logstash, I will be happy to know it!

Thanks again and best regards!

Yanick