Best way to prevent add_fields with %{} when field does not exist

Elastic Stack Logstash
1

I have a lot of csv fields that i'm using a csv filter to parse and sometimes the fields are blank. Right after parsing them, I assign the results to new fields using mutate's add_field. Here is a sample

csv {
separator => "," columns => [ "pid", "name", "dev_contact", "dev_location", "dev_vendor", "dev_family"]
}

mutate {
add_field => {
"id" => "%{pid}"
"name" => "%{name}"
"[dev][contact]" => "%{dev_contact}"
"[dev][location]" => "%{dev_location}"
"[dev][vendor]" => "%{dev_vendor}"
"[dev][family]" => "%{dev_family}"
}
}

Using this method, if there is no entry for any field, such as dev_contact, then my [dev][contact] will contain
%{dev_contact}.

I'm aware that I can check if dev_contact exists before assignment like this:

if [dev_contact]{
mutate {
add_field => { "[dev][contact]" => "%{dev_contact}" }
}
}

but i really do not want to do that for all of my fields since my actual csv has hundreds of fields. Is there a better way to prevent add_field from putting the variable in if nothing exists?

Thanks for any ideas!

2

No, but you could e.g.

  • write a snippet of ruby in a ruby filter to do all the necessary mutations or
  • generate the necessary configuration so it doesn't matter if you have hundreds of conditionals.
3

Thanks, Magnus. I just wanted to verify that I wasn't making things harder than they should be.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.