Subtracting two values of a field

[zaeemmasood]

Hello All,

I have a set up which parses the following line in a log file:

[2022-06-10T19:52:05.017+0000][info][gc             ] GC(35959) Pause Full (Diagnostic Command) 1850M->1140M(2560M) 506.831ms

The resultant gets displayed in Kibana as follows. See excerpt:

I need to subtract the heapDrop values i.e. 1850 -> 1140 and store the resultant in a different field which I could use to create charts from.

My config looks like below:

          if [type] == "tv_gclog_analysis"  {

                        if "start" in [message] { drop{} }
                        if "Full" in [message] {
                                grok {
                                        match => { "message" => '\[%{TIMESTAMP_ISO8601:createdTime}\]\[%{WORD:logLevel}\]+%{GREEDYDATA:message} %{GREEDYDATA:heapDrop}\(%{DATA:maxHeap:int}\) %{GREEDYDATA:timeTaken:float}' }
                        }

                                }
                        else {
                               drop {}
                            }
                        }
                }

Please guide. Do I need to use a ruby filter to make this happen?

[Badger]

If you have the bytes filter installed (it is not bundled by default) you could use

    grok { match => { "heapDrop" => "%{WORD:upper}->%{WORD:lower}" } }
    mutate { replace => { "upper" => "%{upper}B" "lower" => "%{lower}B" } }
    bytes { source => "upper" target => "upper" }
    bytes { source => "lower" target => "lower" }
    ruby { code => 'event.set("delta", event.get("upper").gsub("M", "000000").gsub("G", "000000000").gsub - event.get("lower"))' }

If you cannot or do not want to install it then

    grok { match => { "heapDrop" => "%{WORD:upper}->%{WORD:lower}" } }
    ruby {
        code => '
            lower = event.get("lower")
            upper = event.get("upper")
            if lower and upper
                lower = lower.gsub("M", "000000").gsub("G", "000000000").gsub("T", "000000000000").to_i
                upper = upper.gsub("M", "000000").gsub("G", "000000000").gsub("T", "000000000000").to_i
                event.set("delta", upper - lower)
            end
        '
    }

but there is a big difference between 710000000 and 744488960.

[zaeemmasood]

@Badger

but there is a big difference between 710000000 and 744488960

Hi Badger. Sorry couldn't get this part.

Also I will have to go for the second option as the servers are not open to internet.

[Badger]

@zaeemmasood

Sorry couldn't get this part.

The bytes filter will convert 1 MB to 1024x1024. The gsub will convert 1 MB to 1000x1000. I am saying that those small errors add up surprisingly quickly. The difference is between 710 million bytes and 744 million bytes.

Depending on your use case this may or may not matter. If you are aggregating the volume of garbage collected over an interval then the difference between 710 million and 744 million may be negligible, whereas if you collect 2 GB of garbage in one hour, and 75 GB of garbage in the next, that may tell you something.

[zaeemmasood]

Thanks @Badger as always!