Converting bytes to gb in logstash configuration file

Elastic Stack Logstash
1

Hi All,

I have logstash conf file, where I have field with bytes but when I Load to Kibana I should see new field with bytes coverted to GB.

I tried using ruby filter but it is failing with error "[2017-09-27T15:16:43,456][ERROR][logstash.filters.ruby ] Ruby exception occurred: Direct event field references (i.e. event['field']) have been disabled in favor of using event get and set methods (e.g. event.get('field')). Please consult the Logstash 5.0 breaking changes documentation for more details."

here is my logstash conf file
input {
file {
path => "/apps/mft/ELK/MBXLogFiles/IdentityTestingLog.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["Destination","MailboxName","BPId","MessageId","FileName","Bytes","Date","Time","Action"]
}
grok {
match => ["MailboxName", "^/(?[^/,]+)"]
}
mutate {
convert => { "Bytes" => "integer" }
}
if [Bytes]{
ruby {
code => "event['SizeinGB'] = event['Bytes']/1024/1024/1024"
}
}

}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["192.168.56.105:9200"]
index => "23andme"
document_type => "mbxlogdoctype"
}

}

could some one help me on this

Thanks
Somu

2

Did you consult the Logstash breaking changes documentation as the error message suggests?

https://www.elastic.co/guide/en/logstash/current/breaking-changes.html#_ruby_filter_and_custom_plugin_developers

3

Hi Magnus,

Thank you for answering my question

I updated my conf file as per the documentation link which you shared, this time I didnt see any error but I see SizeinGb filed with value "0"

  "Bytes" => 704202516,
 "@timestamp" => 2017-09-27T20:35:53.479Z,
       "BPId" => "22720029",
   "@version" => "1",
       "host" => "sinode0",
   "SizeinGB" => 0,

here is my updated conf
input {
file {
path => "/apps/mft/ELK/MBXLogFiles/IdentityTestingLog.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["Destination","MailboxName","BPId","MessageId","FileName","Bytes","Date","Time","Action"]
}
grok {
match => ["MailboxName", "^/(?[^/,]+)"]
}
mutate {
convert => { "Bytes" => "integer" }
}
if [Bytes]{
ruby {
code => "event.set('[SizeinGB]',event.get('Bytes')/1024/1024/1024)"
}
}

}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["192.168.56.105:9200"]
index => "23andme"
document_type => "mbxlogdoctype"
}

}

4

could you help me in getting bytes converted to GB..now I dont know what I am missing because I didnt see any error

5

Magnus,

this is intersting multiplication is working i.e..,

ruby {
code => 'event.set("SizeinGB",event.get("Bytes")*2)'
}

but if i want division it is giving zero value

6

704202516 / 1024 / 1024 / 1024 is 0.65583, which gets truncated to zero when doing integer arithmetic. You'll get the desired result if you call to_f on the number of bytes, i.e. replace event.get("Bytes") with event.get("Bytes").to_f.

7

Thank you Magnus. I able to get now

nice to know about to_f

I created visualization with date histogram monthly on x-axis, but some how it is showing too tip on x-axis for every 3 months not for every month though the vertical bars are showing for every month.

I uploaded the image..please see and let me know

VerticalBar
VerticalBar.PNG1739×848 55 KB

8

I'm not sure the x axis label distance is configurable. You should ask in the Kibana group.

9

Thank you Magnus..I will ask in Kibana grop

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.