Prevent %{field_name} in output, for fields not always present

Elastic Stack Logstash
1

Hello,

Basically, the problem is that I need to include fields in my Logstash outputs that aren't always present in the event, but are sometimes present. By not present I mean that the field name is not a defined field in the input at all, not just that the field is defined as having no value. I do not want to use a complex set of hard-to-maintain conditionals to exclude those fields from the output or manually set them as null when they aren't present, and I worry about running a 'gsub' or similar on the final results for fear of removing any %{legitimate_values_that_just_happen_to_look_like_this}.

Ideally I wish I could specify a default value for each %{field_name} when including it in an output or creating a new field that contains it, a la: %{field_name|default('')} ...but I don't think there is such a syntax in the Logstash config.

Here are two examples of this issue...

When the outbound email service our team uses cannot send an email the resulting event from their /events API will sometimes provide a "reason" field that describes the failure (such as when they reject a message based on previous bounces, invalid MX detected before send, etc.). Other times, if they've actually attempted delivery and gotten a failure back from the remote smtp host they will provide a "message" field within the event.

We use an email output in Logstash to notify us of these more mundane failures (simply accepting and handling the obvious case of when this service is down and we really can't email at all). The problem I'm having is that all of the emails from this output have a %{field_name} in one field or the other, because both fields are never (or almost never) present in the same event.

...
Reason: Delivery not attempted - Invalid MX for remote domain.
Message: %{message}
...

or

...
Reason: %{reason}
Message: Remote host said user does not exist
...

.

Also there is a similar situation when processing our application's logs where I create a new field that can be included in a different email output (it sort of re-creates log lines for emailing)...

if "cloned_for_multiline_error_notification" in [tags] {
    # add the combined field that would actually get emailed out if this group of events gets tagged with "bad"
    if [type] == "symfony2json" {
        mutate {
            add_field => { "combined_for_email" => "%{[extra][clientip]} %{[datetime][date]} %{level_name} %{message} %{[content][exception]} <br/>" }
        }
    }
}

After this section, events are individually tagged with bad_event if we detect them to be so. They then go through a multiline filter and are either emailed out (if the merged event contains the bad_event tag), or simply dropped.

The problem here is that the resulting email shows bad events in the context of a few events that came before and after them, but usually some of the contextual events are not "bad". So those "good" lines have %{[content][exception]} in them because that field is not defined. Here are an examples of "good" and "bad" events from an email...

,51.142.157.26 2015-09-25 01:17:29 INFO Matched route "application_front_end_item" (parameters: "_controller": "Application\FrontEnd\Controller::itemAction") %{[context][exception]}
,#bad_event 51.142.157.26 2015-09-25 01:17:29 ERROR Uncaught exception Cannot Fetch Record - Not Found

Any ideas?

Thanks for reading,
Mark

2

It may be possible to use conditionals to do this, but you'd have to do it for known field names:

if ![field_name] {
  mutate {
    add_field => { "field_name" => "The default value I'd like to have here" }
  }
}
3

That's what I was afraid of =/, but good to know I'm not simply ignorant of a better way. Fortunately I do have the field names available in advance for the cases I've seen so far, it's just a little verbose.

Thanks for the input,
Mark