# Parse numbers from Storage units in logs

**URL:** <https://discuss.elastic.co/t/parse-numbers-from-storage-units-in-logs/218400>\
**Category:** Logstash\
**Created:** [February 7, 2020, 8:21pm UTC](https://discuss.elastic.co/t/parse-numbers-from-storage-units-in-logs/218400 "2020-02-07T20:21:31Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![nmoham](https://avatars.discourse-cdn.com/v4/letter/n/77aa72/32.png) [@nmoham](https://discuss.elastic.co/u/nmoham)\
**Post date:** [February 7, 2020, 8:21pm UTC](https://discuss.elastic.co/t/parse-numbers-from-storage-units-in-logs/218400/1 "2020-02-07T20:21:32Z")

</div>

Extract the NUMBER from logs with storage sizes.

Sample Logs -

physical.memory.total=31.4G, physical.memory.free=18.8G, swap.space.total=7.8G, swap.space.free=7.8G, heap.memory.used=4.5G, heap.memory.free=536.5M, heap.memory.total=5.0G, heap.memory.max=5.0G, native.memory.used=8.2M, native.memory.free=3.5G, native.memory.total=64.0M, native.memory.max=3.5G, native.meta.memory.used=80.0M, native.meta.memory.free=432.0M

Trying to extract the NUMBER for each of these fields. The challenge is it can either be G/M units .  
What would be the best way to achieve this ? appreciate any help or guidance .

Thank you

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 7, 2020, 8:48pm UTC](https://discuss.elastic.co/t/parse-numbers-from-storage-units-in-logs/218400/2 "2020-02-07T20:48:35Z")

</div>

I have done something similar in the past using mutate to replace K with 000, M with 000000 etc, but it would be more accurate to do

```
    kv { field_split => "," value_split => "=" trim_key => " " target => "[@metadata][metrics]" }
    ruby {
        code => '
            metrics = event.get("[@metadata][metrics]")
            if metrics.is_a? Hash
                metrics.each { |k, v|
                    v.scan(/(.*)([KMG])$/) { |m|
                        case m[1]
                        when "K"
                            multiplyBy = 1024
                        when "M"
                            multiplyBy = 1024 * 1024
                        when "G"
                            multiplyBy = 1024 * 1024 * 1024
                        end
                        event.set(k, m[0].to_f * multiplyBy)
                    }
                }
            end
        '
    }
```

---

<div class="post-metadata">

**Author:** ![nmoham](https://avatars.discourse-cdn.com/v4/letter/n/77aa72/32.png) [@nmoham](https://discuss.elastic.co/u/nmoham)\
**Post date:** [February 7, 2020, 10:31pm UTC](https://discuss.elastic.co/t/parse-numbers-from-storage-units-in-logs/218400/3 "2020-02-07T22:31:06Z")

</div>

Thanks Badger.

I gave it a try , worked perfectly; but it dropped many other fields like "load.process=0.00%, load.system=0.01%, load.systemAverage=1.00%". I see some errors in the logs, probably because of other fields extracted using the KV Filter;

Here are the sample logs -

2020-02-04 16:18:56,783 INFO Log4jFactory$Log4jLogger [10.190.100.222]:5701 [Drysdale-PROD-SC9] [3.7.6] Received auth from Connection[id=26876, /10.190.100.222:5701-\>/10.190.110.202:56584, endpoint=null, alive=true, type=CSHARP\_CLIENT], successfully authenticated, principal : ClientPrincipal{uuid='d7d8b718-ed75-4cc3-b51a-c620bb082255', ownerUuid='058720ad-7b35-40f6-8978-bd9cf7e286ec'}, owner connection : true, client version : null

2020-02-04 16:15:27,519 INFO Log4jFactory$Log4jLogger [10.190.100.222]:5701 [Drysdale-PROD-SC9] [3.7.6] processors=8, physical.memory.total=31.4G, physical.memory.free=18.8G, swap.space.total=7.8G, swap.space.free=7.8G, heap.memory.used=4.5G, heap.memory.free=536.5M, heap.memory.total=5.0G, heap.memory.max=5.0G, heap.memory.used/total=89.51%, heap.memory.used/max=89.51%, native.memory.used=8.2M, native.memory.free=3.5G, native.memory.total=64.0M, native.memory.max=3.5G, native.meta.memory.used=80.0M, native.meta.memory.free=432.0M, native.meta.memory.percentage=90.75%, minor.gc.count=18605, minor.gc.time=121136ms, major.gc.count=0, major.gc.time=0ms, load.process=0.00%, load.system=0.01%, load.systemAverage=1.00%, thread.count=69, thread.peakCount=229, cluster.timeDiff=4083, event.q.size=0, executor.q.async.size=0, executor.q.client.size=0, executor.q.query.size=0, executor.q.scheduled.size=0, executor.q.io.size=0, executor.q.system.size=0, executor.q.operations.size=0, executor.q.priorityOperation.size=0, operations.completed.count=351751533, executor.q.mapLoad.size=0, executor.q.mapLoadAllKeys.size=0, executor.q.cluster.size=0, executor.q.response.size=0, operations.running.count=0, operations.pending.invocations.percentage=0.00%, operations.pending.invocations.count=1, proxy.count=0, clientEndpoint.count=231, connection.active.count=232, client.connection.count=231, connection.count=1

Is it possible that we only affect memory related fields that contain?

my current Logstash conf looks like this -

```
  if [name] == "hazel_logs"
    {
    grok {
        match => {"message" => ["(?m)%{TIMESTAMP_ISO8601:logtime}%{SPACE}%{LOGLEVEL:log.level}%{SPACE}\[%{HOSTNAME:hazel.server}\]\:%{NUMBER:hazelcast.port}%{SPACE}\[%{WORD:hazelcast.env}\]%{SPACE}\[(?<hazelcast.version>[\d+.]+)\]+%{SPACE}%{GREEDYDATA:msgbody}","(?m)%{TIMESTAMP_ISO8601:logtime}%{SPACE}%{LOGLEVEL:log.level}%{SPACE}%{DATA:logger}%{SPACE}\[%{IPORHOST:hazel.server}\]\:%{NUMBER:hazelcast.port}%{SPACE}\[%{DATA:hazelcast.env}\]%{SPACE}\[(?<hazelcast.version>[\d+.]+)\]+%{SPACE}%{GREEDYDATA:msgbody}","(?m)%{TIMESTAMP_ISO8601:logtime}%{SPACE}%{LOGLEVEL:log.level}%{SPACE}%{GREEDYDATA:msgbody}"] } }

            kv {
                    source => "msgbody"
                    value_split => "="
                    allow_duplicate_values => false
                    remove_char_key => "\s\n{\[\]"
                    remove_char_value => "\s\n{},'\[\]%"
                    transform_key => "lowercase"
                    recursive => "true"
		target => "[@metadata][metrics]"
                    #trim_value => ",'%"
                    #whitespace => "strict"
                    }
             grok {
                    patterns_dir => "/etc/logstash/patterns"
                    break_on_match => "false"
                    match => {
                    "msgbody" => ["Reason:(?<error.details>.[^\[\,\n?]+)\s?", "type=%{DATA:hazelcast.client.type}\]", "minor.gc.time=%{NUMBER:minor.gc.time.ms:int}", "major.gc.time=%{NUMBER:major.gc.time.ms:int}" ]
                            }
                    }
	date {
    		locale => "en"
		match => ["logtime", "YYYY-MM-dd HH:mm:ss,SSS"]
		timezone => "America/Los_Angeles"
        		target => "@timestamp"
    		}

            ruby {
                    code => "

                    event.to_hash.keys.each { |k,v|
                    if ( k.end_with?('size') || k.end_with?('count'))
                            event.set(k, event.get(k).to_i)
                    elsif ( k.start_with?('load') || k.start_with?('processors'))
                            event.set(k, event.get(k).to_f)
		else
			event.set(k, event.get(k))
                    end
                    }
                    "
            }

```

############ Here ############

```
ruby {
    code => '
        metrics = event.get("[@metadata][metrics]")
        if metrics.is_a? Hash
            metrics.each { |k, v|
                v.scan(/(.*)([KMG])$/) { |m|
                    case m[1]
                    when "K"
                        multiplyBy = 1024
                    when "M"
                        multiplyBy = 1024 * 1024
                    when "G"
                        multiplyBy = 1024 * 1024 * 1024
                    end
                    event.set(k, m[0].to_f * multiplyBy)
                }
            }
        end
    '
}

```

########### Here #############

```
        mutate {
    	        replace => { "type" => "hazelcast" }
    	        remove_field => ["msgbody", "logtime", "logger", "thread"]
            	}
	prune
		{
		blacklist_names => ["serversocketaddr*", "[0-9]+", "gc.time$" ]
		}
}

```

}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 8, 2020, 12:44am UTC](https://discuss.elastic.co/t/parse-numbers-from-storage-units-in-logs/218400/4 "2020-02-08T00:44:45Z")

</div>

> [@nmoham](#):
>
> I gave it a try , worked perfectly; but it dropped many other fields like "load.process=0.00%,

OK, so instead of using

```
v.scan(/(.*)([KMG])$/) { |m|

```

use

```
matches = v.scan(/(.*)([KMG])$/)

```

then if matches is nil do a straight event.set(k, v), otherwise execute the case statement and the event.set I gave.

---

<div class="post-metadata">

**Author:** ![nmoham](https://avatars.discourse-cdn.com/v4/letter/n/77aa72/32.png) [@nmoham](https://discuss.elastic.co/u/nmoham)\
**Post date:** [February 10, 2020, 12:27am UTC](https://discuss.elastic.co/t/parse-numbers-from-storage-units-in-logs/218400/5 "2020-02-10T00:27:00Z")

</div>

> [@Badger](#):
>
> matches = v.scan(/(.\*)([KMG])$/)

hi @Badger,

So, I tried the suggested method and now it is dropping all the fields completely discovered from the KV filter.

Exceptions in Logstash logs -

[2020-02-09T16:17:21,451][ERROR][logstash.filters.ruby][main] Ruby exception occurred: undefined method `match' for ["CSHARP_CLIENT", "CSHARP_CLIENT."]:Array [2020-02-09T16:17:21,451][ERROR][logstash.filters.ruby][main] Ruby exception occurred: undefined method `match' for ["CSHARP\_CLIENT", "CSHARP\_CLIENT."]:Array  
[2020-02-09T16:17:23,973][ERROR][logstash.filters.ruby][main] Ruby exception occurred: undefined method `[]' for nil:NilClass [2020-02-09T16:17:26,152][ERROR][logstash.filters.ruby][main] Ruby exception occurred: undefined method `' for nil:NilClass

Ruby Code -

ruby {  
code =\> "  
metrics = event.get('[@metadata][metrics]')  
if metrics.is\_a? Hash  
metrics.each { |k, v|  
matches = v.match(/(.\*)([KMG])$/)  
case  
when nil  
event.set(k,v)  
when 'K'  
multiplyBy = 1024  
when 'M'  
multiplyBy = 1024 \* 1024  
when 'G'  
multiplyBy = 1024 \* 1024 \* 1024  
end  
event.set(k, matches[0].to\_f \* multiplyBy)  
}  
end  
"  
}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 10, 2020, 2:05am UTC](https://discuss.elastic.co/t/parse-numbers-from-storage-units-in-logs/218400/6 "2020-02-10T02:05:49Z")

</div>

Did you read the post in which I responded to you?

---

<div class="post-metadata">

**Author:** ![nmoham](https://avatars.discourse-cdn.com/v4/letter/n/77aa72/32.png) [@nmoham](https://discuss.elastic.co/u/nmoham)\
**Post date:** [February 10, 2020, 2:08am UTC](https://discuss.elastic.co/t/parse-numbers-from-storage-units-in-logs/218400/7 "2020-02-10T02:08:13Z")

</div>

do you mean this ?

use

```auto
matches = v.scan(/(.*)([KMG])$/)

```

"then if matches is nil do a straight event.set(k, v), otherwise execute the case statement and the event.set I gave."

---

<div class="post-metadata">

**Author:** ![nmoham](https://avatars.discourse-cdn.com/v4/letter/n/77aa72/32.png) [@nmoham](https://discuss.elastic.co/u/nmoham)\
**Post date:** [February 10, 2020, 2:09am UTC](https://discuss.elastic.co/t/parse-numbers-from-storage-units-in-logs/218400/8 "2020-02-10T02:09:58Z")

</div>

I did change the code to -

```auto
ruby {
code => "
metrics = event.get('[@metadata][metrics]')
if metrics.is_a? Hash
metrics.each { |k, v|
matches = v.match(/(.*)([KMG])$/)
case
when nil
event.set(k,v)
when 'K'
multiplyBy = 1024
when 'M'
multiplyBy = 1024 * 1024
when 'G'
multiplyBy = 1024 * 1024 * 1024
end
event.set(k, matches[0].to_f * multiplyBy)
}
end
"
}

```

Doing something wrong here ?

---

<div class="post-metadata">

**Author:** ![nmoham](https://avatars.discourse-cdn.com/v4/letter/n/77aa72/32.png) [@nmoham](https://discuss.elastic.co/u/nmoham)\
**Post date:** [February 11, 2020, 5:48pm UTC](https://discuss.elastic.co/t/parse-numbers-from-storage-units-in-logs/218400/9 "2020-02-11T17:48:05Z")

</div>

Tried changing from using case to regular if conditions. Still see errors and no results.  
Appreciate any extra directions.

Thanks

---

<div class="post-metadata">

**Author:** ![nmoham](https://avatars.discourse-cdn.com/v4/letter/n/77aa72/32.png) [@nmoham](https://discuss.elastic.co/u/nmoham)\
**Post date:** [February 23, 2020, 9:23pm UTC](https://discuss.elastic.co/t/parse-numbers-from-storage-units-in-logs/218400/10 "2020-02-23T21:23:44Z")

</div>

Ended up using the logstash bytes filter plugin, was little more work, but worked as expected.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 22, 2020, 9:23pm UTC](https://discuss.elastic.co/t/parse-numbers-from-storage-units-in-logs/218400/11 "2020-03-22T21:23:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.