Removing specific fields

[Nikolay_Petrov (raba)]

Hi,
I'm trying to remove fields like this "*_bytes" using logstash filter, without any success so far:

{
...
"mountpoints": {
"/soft": {
"size_bytes": "21003628544",
"available_bytes": "20957556736",
"used_bytes": "46071808",
},
"/dev/shm": {
"size_bytes": "521474048",
"available_bytes": "521474048",
"used_bytes": "0",
},
"/": {
"size_bytes": "8319852544",
"available_bytes": "5288181760",
"used_bytes": "3031670784",
}
},
"disks": {
"xvdb": {
"size_bytes": "21474836480"
},
"xvda": {
"size_bytes": "8589934592"
}
},
"partitions": {
"/dev/xvda1": {
"size_bytes": "8588886016",
}
},
...
}

my logstash logstash filter:

...
ruby {
code => "
event.to_hash.keys.each { |k|
if k.end_with?('bytes')
event.remove(k)
end
}
"
}

prune {
    interpolate => true
    blacklist_names =>  [ "^.*bytes$", "&{}>;,", "[^.*][.*][.*_bytes$]" ]
}

mutate {
    remove_field => [ ".*bytes", "^.*[.*bytes]$" ]
 }

...
i'm not sure what i missed into the regexp or any other filter that can be used.

[magnusbaeck (Magnus Bäck)]

The prune filter doesn't work with nested fields (https://github.com/logstash-plugins/logstash-filter-prune/issues/12) and your ruby snippet only processes the keys at the top level of the event. You'd have to write Ruby code that recurses into the subhashes. I don't have time to provide an example of that, but since it's not a Logstash-specific problem I imagine you'll find a solution at e.g. StackOverflow.