Logstash filter for varying number of disks info in the logs

Elastic Stack Logstash
1

Hello,

i have a couple of log lines which go like this , that have varying number of disks

1628868082.953|7ec93a454940| disk-usage: [1]=46% [2]=44% [3]=52% [4]=52% [5]=46% [6]=40% [7]=45% [8]=48% 
1628868082.953|7ec93a454940| disk-usage: [1]=38% [2]=10% 
1628868082.953|7ec93a454940| disk-usage: [1]=46% [2]=44% [3]=52% [4]=52% [5]=46% [6]=40% [7]=45% [8]=48% [9]=15% [10]=28%

I tried writing a grok filter with multiple patterns , but hasnt worked.

match => {"message" =>["%{NUMBER:epoch_time}%{NOTSPACE:ThreadID} %{WORD:cache}-%{WORD:cache}: %{NOTSPACE:cache}=%{NUMBER:Disk_1}% %{NOTSPACE:cache}=%{NUMBER:Disk_2}% %{NOTSPACE:cache}=%{NUMBER:Disk_3}% %{NOTSPACE:cache}=%{NUMBER:Disk_4}% %{NOTSPACE:cache}=%{NUMBER:Disk_5}% %{NOTSPACE:cache}=%{NUMBER:Disk_6}% %{NOTSPACE:cache}=%{NUMBER:Disk_7}% %{NOTSPACE:cache}=%{NUMBER:Disk_8}%","%{NUMBER:cache_time}%{NOTSPACE:ThreadID} %{WORD:cache}-%{WORD:cache}: %{NOTSPACE:cache}=%{NUMBER:Disk_1}% %{NOTSPACE:cache}=%{NUMBER:Disk_2}%"]}

Can someone help?

2

I would not use grok for that. I would use dissect and kv.

    dissect { mapping => { "message" => "%{[@metadata][ts]}|%{}| disk-usage: %{[@metadata][restOfLine]}" } }
    date { match => [ "[@metadata][ts]", UNIX ] }
    kv { trim_value => "%" }

will produce

"@timestamp" => 2021-08-13T15:21:22.953Z,
         "1" => "46",
         "2" => "44",
         "3" => "52",

etc. I would then be tempted to either

    ruby {
        code => '
            diskUsage = []
            event.to_hash.each { |k, v|
                if k =~ /^\d+$/
                    diskUsage[k.to_i] = v.to_i;
                end
                event.remove(k)
            }
            diskUsage.shift
            event.set("diskUsage", diskUsage)
        '
    }

to get

 "diskUsage" => [
    [0] 46,
    [1] 44,
    [2] 52,
    [3] 52,
    [4] 46,
    [5] 40,
    [6] 45,
    [7] 48
]

or

    ruby {
        code => '
            diskUsage = []
            event.to_hash.each { |k, v|
                if k =~ /^\d+$/
                    event.set("diskUsage#{k.to_i}", v.to_i)
                end
                event.remove(k)
            }
        '
    }

to get

 "diskUsage5" => 46,
 "diskUsage4" => 52,
 "diskUsage7" => 45,

etc.

1 Like
3

Thanks a lot @Badger :slight_smile: is there some documentation on how rube code works on logstash ? i'd like to understand it

4

Not that I know of.

5

okay :slight_smile: i'm using dissect for the first time. so i was wondering if "%{}" can be used for empty spaces ?

6

%{} matches anything, it consumes everything up to the next delimiter.

1 Like
7

Thanks a ton @Badger :slight_smile: . The kv + dissect worked perfectly

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.