# Logstash Remove Field Output Issue

**URL:** <https://discuss.elastic.co/t/logstash-remove-field-output-issue/72959>\
**Category:** Logstash\
**Created:** [January 26, 2017, 10:21pm UTC](https://discuss.elastic.co/t/logstash-remove-field-output-issue/72959 "2017-01-26T22:21:50Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [January 26, 2017, 10:21pm UTC](https://discuss.elastic.co/t/logstash-remove-field-output-issue/72959/1 "2017-01-26T22:21:50Z")

</div>

I was cleaning up some code that I use to generate emails for alerts based on some criteria in Logstash.

These alerts send an email as well as create a new document for insertion into Elasticsearch. This gives a historial record of any alerts generated.

When I added some mutate lines to my Logstash config to remove the unused fields AFTER the email output but BEFORE the Elasticsearch output, the emails basically came across as blank.

All I am trying to do, is remove the extra fields before the log gets inserted into Elasticsearch. But it seems, that even though the remove fields config is AFTER the email output config everything breaks.

Am I missing something?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 27, 2017, 8:33am UTC](https://discuss.elastic.co/t/logstash-remove-field-output-issue/72959/2 "2017-01-27T08:33:15Z")

</div>

All filters are processed before the outputs. All outputs see the same events (but you can choose which outputs get which events). Two immediate options:

- Store the email-only fields under the `@metadata` field (which isn't included in what's sent to ES).
- Use a clone filter to splice each event in two. Then you can do whatever mutations you want and route the clone and the original event to different outputs.

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [January 27, 2017, 2:50pm UTC](https://discuss.elastic.co/t/logstash-remove-field-output-issue/72959/3 "2017-01-27T14:50:45Z")

</div>

So,  
Part of what I do is, use a IF statement to check if certain fields have data. Based on the true/false outputs of those checks I build out a section of HTML based code.

When I tried to store that code as part of a metadata field, nothing came across in my email output.

So then I built regular fields to store the sections of code.

I don't want to use non-supported plugins, if at all possible. Clone wasn't on the list, the last time I looked.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 27, 2017, 2:52pm UTC](https://discuss.elastic.co/t/logstash-remove-field-output-issue/72959/4 "2017-01-27T14:52:16Z")

</div>

> When I tried to store that code as part of a metadata field, nothing came across in my email output.

That's surprising. What did your config look like?

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [January 27, 2017, 3:01pm UTC](https://discuss.elastic.co/t/logstash-remove-field-output-issue/72959/5 "2017-01-27T15:01:05Z")

</div>

I am not using this piece of code right now, for obvious reasons, but here it is:

I check to see if the hostname field was populated:

```
if [hostname] !~ /.+/ {
  mutate { add_field => { "hostname" => "Unknown" } }
}

```

This section, merely built field "host\_info" with a section of HTML code:

```
mutate { add_field => { "[@metadata][host_info]" => "<tr bgcolor=#ffffff><td width=20%><font size=2 face=Verdana>Host Info</td><td><font size=2 face=Verdana>%{hostname} (%{host})</td></tr>" } }

```

The output section, in the email plugin looked like this:

```
output {
  email {
    via => "smtp"
    domain => "#####"
    address => "#####"
    from => "%{smtp_from}"
    to => "%{smtp_to}"
    subject => "%{subject_line}"
    htmlbody => "
      <html>
       <body>
        <table border=1 cellspacing=1 cellpadding=5 bgcolor=#00438c width=100%>
         <thead bgcolor=#00438c>
          <tr>
           <td colspan=2><font color=white size=3 face=Verdana>Log Information</font></td>
          </tr>
         </thead>
        <tbody>
          %{device_info}
          %{host_info}
          %{user_info}
          %{log_source_info}
          %{fingerprint_info}
         </tbody>
        <tfoot>
         %{log_message}
        </tfoot>
       </table>
      </body>
     </html>
    "
  }
}
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 27, 2017, 3:14pm UTC](https://discuss.elastic.co/t/logstash-remove-field-output-issue/72959/6 "2017-01-27T15:14:57Z")

</div>

Okay, but that body doesn't reference any `@metadata` fields, yet you say you ended up with nothing in the email that was sent?

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [January 27, 2017, 3:28pm UTC](https://discuss.elastic.co/t/logstash-remove-field-output-issue/72959/7 "2017-01-27T15:28:18Z")

</div>

In the output, the host\_info field was a meta field.

`[@metadata][host_info]`

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 27, 2017, 6:34pm UTC](https://discuss.elastic.co/t/logstash-remove-field-output-issue/72959/8 "2017-01-27T18:34:11Z")

</div>

...and you listed it as `%{[@metadata][host_info]}` in the `htmlbody` option? And it wasn't expanded?

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [January 27, 2017, 6:51pm UTC](https://discuss.elastic.co/t/logstash-remove-field-output-issue/72959/9 "2017-01-27T18:51:55Z")

</div>

No, I sure did not. I only referenced the field name like this %{host\_info}.

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [January 27, 2017, 6:59pm UTC](https://discuss.elastic.co/t/logstash-remove-field-output-issue/72959/10 "2017-01-27T18:59:09Z")

</div>

Give me a few and I will try that again.

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [January 27, 2017, 11:02pm UTC](https://discuss.elastic.co/t/logstash-remove-field-output-issue/72959/11 "2017-01-27T23:02:56Z")

</div>

Magnus, that worked!!!! Thank you!!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 24, 2017, 11:03pm UTC](https://discuss.elastic.co/t/logstash-remove-field-output-issue/72959/12 "2017-02-24T23:03:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.