Logstash filter for varying number of disks info in the logs

Elastic Stack Logstash
2

I would not use grok for that. I would use dissect and kv.

    dissect { mapping => { "message" => "%{[@metadata][ts]}|%{}| disk-usage: %{[@metadata][restOfLine]}" } }
    date { match => [ "[@metadata][ts]", UNIX ] }
    kv { trim_value => "%" }

will produce

"@timestamp" => 2021-08-13T15:21:22.953Z,
         "1" => "46",
         "2" => "44",
         "3" => "52",

etc. I would then be tempted to either

    ruby {
        code => '
            diskUsage = []
            event.to_hash.each { |k, v|
                if k =~ /^\d+$/
                    diskUsage[k.to_i] = v.to_i;
                end
                event.remove(k)
            }
            diskUsage.shift
            event.set("diskUsage", diskUsage)
        '
    }

to get

 "diskUsage" => [
    [0] 46,
    [1] 44,
    [2] 52,
    [3] 52,
    [4] 46,
    [5] 40,
    [6] 45,
    [7] 48
]

or

    ruby {
        code => '
            diskUsage = []
            event.to_hash.each { |k, v|
                if k =~ /^\d+$/
                    event.set("diskUsage#{k.to_i}", v.to_i)
                end
                event.remove(k)
            }
        '
    }

to get

 "diskUsage5" => 46,
 "diskUsage4" => 52,
 "diskUsage7" => 45,

etc.

1 Like