"mutate gsub" on each element of an array

Elastic Stack Logstash
1

Hi, I'm pretty new to logstash and have a problem I'm not able to solve by trying or searching the web (didn't find any similar questions/solutions yet :frowning: )

I'm using logstash to extract MAC addresses from log lines with the %{MAC} grok pattern. Since I want to use Kibana to filter for these MAC addresses, I put them in a field with %{MAC:mac_address}.

Unfortunately, the logs with the MAC addresses come from different systems which use different syntax for the MAC addresses, like Windows, Linux, Cisco. So it might happen that when to different systems log MAC addresses they are stored as "0123.4567.89ab" when a Cisco System logs this MAC address and "01-23-45-67-89-AB" when a Windows systems logs this MAC address.

My idea was to "normalize" these MAC addresses using the mutate filter before indexing them in Elasticsearch by making them all lowercase and removing all special chars (in fact, I'm converting them to "Cisco"-style which I find most pleasing).

# Normalize field "mac_address"
if [mac_address] =~ /.+/ {
    # mutate lowercase returns "nil" if string is already in lowercase...
    if [mac_address] =~ /[A-F]/ {
        mutate {
            lowercase => [ "mac_address" ]
        }
    }
    # Always format MAC address in Cisco display syntax => 0123.4567.89ab
    mutate {
        gsub => [
            "mac_address", "[^0-9a-f]", "",
            "mac_address", "([0-9a-f]{4})([0-9a-f]{4})([0-9a-f]{4})", "\1.\2.\3"
        ]
    }
}

This works as expected when there is only one MAC address in the log line. When the log line contains multiple MAC addresses, the field "mac_address" is an array, and the gsub filter doesn't work anymore. I know that this is documented on the mutate page under https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-gsub

I have the feeling that a solution would be a ruby filter, but I'm not very used to Ruby programming, and my solution needs to do some regex replacement on each element of an array. I simpy don't know how to code that in Ruby.

Does anyone has an idea how to solve this?

Thanks in advance for your support!

2

Hi, the problem of normalizing bothered me, so I searched the web and read some documentation. My current solution looks like this:

# Normalize field "mac_address"
if [mac_address]
{
    ruby
    {
        code => "
            mac = event['mac_address']

            if mac.kind_of?(String)
                tmp = [ mac ]
                mac = tmp
            end

            tmp = Array.new
            mac.each{ |m| tmp.push(m
                .downcase
                .gsub(/[^0-9a-f]/, '')
                .gsub(/([0-9a-f]{4})([0-9a-f]{4})([0-9a-f]{4})/, '\1.\2.\3')
            )}

            if tmp.length > 1
                event['mac_address'] = tmp
            elsif tmp.length == 1
                event['mac_address'] = tmp[0]
            else
                event.remove('mac_address')
            end
        "
    }
}

This code turns the field 'mac_address' to an array, copies each element to a new array while normalizing the string and writes the data back to the 'mac_address' field as an array or -- if only one element is in the array -- as a string.

I don't think that's the perfect solution, but it works. Hints for optimizing the ruby code appreciated.

Regards,
Thorsten