As we are now collecting access logs of our Checkpoint Firewalls, we are facing the issue of horrible inconsistent key values (so the name of the keys itself ). They are formated in different ways but I was able to get them into a consistent format whith logstash but there is still one issue: Some keys have a whitespace INSIDE the key name.
I need a way to replace whitespaces inside key names with underscores. It seems like the trim_key option inside the kv filter was able to do this in an earlier version, but someone thought it's not a good idea - see: https://github.com/logstash-plugins/logstash-filter-kv/issues/10
Unfortunately what the "trim_key" option did in earlier kv filter version is exactly what i would need now. Anyone have an idea how I can accomplish this?
Well that is something I am already doing > Just renaming the key via mutate. So for example "destination dns hostname" becomes "destination_dns_hostname".
But as you have said that is only working if you know the names of all keys. As I'm using kv filter I do not know all key names in advance. New keys can popup everytime so maintaining a "list" is not really a way which scales.
If you only need to get the field consistent (because whitespace use varies), you could also remove the whitespaces altogether with the remove_char_key option in your kv filter:
Contrary to trim option, all characters are removed from the key, whatever their position.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
). They are formated in different ways but I was able to get them into a consistent format whith logstash but there is still one issue: Some keys have a whitespace INSIDE the key name.
