Dealing with default values and different outputs

[UnitedMarsupials]

Various data-fields supplied by our CDN-providers sometimes contain dashes (a.k.a. hyphens) to indicate the particular piece of data being unavailable.

We need to loop through all (well, most) of the fields and remove such empty ones so Elastic never sees them. Such looping can be accomplished with the ruby-filter or even in Jinja (which we use to generate logstash config-files) -- although, perhaps, a better solution exists.

However, our other output is a flat tab-separated file, which still needs something -- referring to a field, that does not exist in an event will output a nasty %{fieldname} instead of an empty string.

Yet another output is Graphite, which sometimes needs a 0 instead of a missing data-item...

What's the best way to deal with such a situation -- presenting the same event differently to the different outputs?

[magnusbaeck (Magnus Bäck)]

As long as the fields in question are top-level fields you can use the prune filter to remove fields containing e.g. a lone hyphen.

As to your general question events are presented to outputs in the same way. The best you can do is probably to use the clone filter to splice events into multiple copies and then modify the copies in different ways.