Logstash Remove Field Output Issue

Elastic Stack Logstash
1

I was cleaning up some code that I use to generate emails for alerts based on some criteria in Logstash.

These alerts send an email as well as create a new document for insertion into Elasticsearch. This gives a historial record of any alerts generated.

When I added some mutate lines to my Logstash config to remove the unused fields AFTER the email output but BEFORE the Elasticsearch output, the emails basically came across as blank.

All I am trying to do, is remove the extra fields before the log gets inserted into Elasticsearch. But it seems, that even though the remove fields config is AFTER the email output config everything breaks.

Am I missing something?

2

All filters are processed before the outputs. All outputs see the same events (but you can choose which outputs get which events). Two immediate options:

  • Store the email-only fields under the @metadata field (which isn't included in what's sent to ES).
  • Use a clone filter to splice each event in two. Then you can do whatever mutations you want and route the clone and the original event to different outputs.
1 Like
3

So,
Part of what I do is, use a IF statement to check if certain fields have data. Based on the true/false outputs of those checks I build out a section of HTML based code.

When I tried to store that code as part of a metadata field, nothing came across in my email output.

So then I built regular fields to store the sections of code.

I don't want to use non-supported plugins, if at all possible. Clone wasn't on the list, the last time I looked.

4

When I tried to store that code as part of a metadata field, nothing came across in my email output.

That's surprising. What did your config look like?

5

I am not using this piece of code right now, for obvious reasons, but here it is:

I check to see if the hostname field was populated:

if [hostname] !~ /.+/ {
  mutate { add_field => { "hostname" => "Unknown" } }
}

This section, merely built field "host_info" with a section of HTML code:

mutate { add_field => { "[@metadata][host_info]" => "<tr bgcolor=#ffffff><td width=20%><font size=2 face=Verdana>Host Info</td><td><font size=2 face=Verdana>%{hostname} (%{host})</td></tr>" } }

The output section, in the email plugin looked like this:

output {
  email {
    via => "smtp"
    domain => "#####"
    address => "#####"
    from => "%{smtp_from}"
    to => "%{smtp_to}"
    subject => "%{subject_line}"
    htmlbody => "
      <html>
       <body>
        <table border=1 cellspacing=1 cellpadding=5 bgcolor=#00438c width=100%>
         <thead bgcolor=#00438c>
          <tr>
           <td colspan=2><font color=white size=3 face=Verdana>Log Information</font></td>
          </tr>
         </thead>
        <tbody>
          %{device_info}
          %{host_info}
          %{user_info}
          %{log_source_info}
          %{fingerprint_info}
         </tbody>
        <tfoot>
         %{log_message}
        </tfoot>
       </table>
      </body>
     </html>
    "
  }
}
6

Okay, but that body doesn't reference any @metadata fields, yet you say you ended up with nothing in the email that was sent?

7

In the output, the host_info field was a meta field.

[@metadata][host_info]

8

...and you listed it as %{[@metadata][host_info]} in the htmlbody option? And it wasn't expanded?

9

No, I sure did not. I only referenced the field name like this %{host_info}.

10

Give me a few and I will try that again.

11

Magnus, that worked!!!! Thank you!!

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.