Data Type Conversion for bytes

Elastic Stack Logstash
#1

Hi, I am importing a csv file and need to convert the fields that contain numbers to float to compute them in Kibana. However the bytes fields that contain volume of data should in Kibana be recognized as such. How can I do this, here is the current filter configuration I am using:

filter {
csv {
columns => [
"BytesDownloaded",
"BytesUploaded",
]
separator => ","
remove_field => ["message"]
}
mutate {
convert => { "BytesDownloaded" => "float" }
convert => { "BytesUploaded" => "float" }
}
}
}

#2

Does this not work? What is the field mapped as in ES?

#3

They are seen as numbers only

#4
  • Why are you converting the fields to floats? Byte counts are discrete and should be integers.
  • How are the fields currently mapped in ES? Use the get mapping API. I have no idea what you mean by "They are seen as numbers only".
#5

Apologies for not being clear, let me try and elaborate. The objective is the following:
I have a CSV file that I would like to use in Kibana and when computing the volumes in bytes that the dashboard automatically computes values to MBytes, Gytes, etc. as applicable. As per the filter on top is what I am currently using, I also tried integers however when in Kibana dashboard it shows in bytes only. Here is an example:

In Kibana the type is number.

In Elastic search it is mapped as the following:
"Bytes_of_Downlink_Data_Packets_Received_Interface(Byte)":{"type":"double"}

I am using the default logstash indexing:
output {
elasticsearch {
hosts => ["localhost:9200"]

index => "logstash-%{+MM.YYYY}"

}
}

#6

Currently the only way to do that is to create a scripted field and divide by 1024^N. Or do the same thing via a ruby filter in LS before it hits ES.

I think KB 5 will allow you to set a field modifier so you can dynamically "convert" such things.

#7