Multiple outputs for same log entry

[duncOps (Duncan Wraight)]

Hi folks,

I've recently written a couple of email module output blocks to notify Teams / PagerDuty when errors occur in our live environment.

My Logstash output looks like this:

output {
  if "PROD" in [log.environment] {
    # All log.error messages spammed to Alerts & Notifications channel
    if "output-teams" in [tags] {
      # Some of our logs are based on REQUEST/RESPONSE and just include a status. These should always be 500+ errors
      if [app.req.destination] and [app.req.method] {
        email {
          << email stuff >>
        }
      }
      # Our newer logs however use "app.logText" and "app.error.messages/response"
      else if [app.logText] {
        email {
          << email stuff >>
        }
      }
    }

    # 500 errors logged out to PagerDuty
    if "output-pagerduty" in [tags] {
      email {
        << email stuff >>
      }
    }
  }

  elasticsearch {
    hosts => "10.128.x.x"
    index => "xxx-%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }

  # comment this out on Pre-Prod and Prod environments
  stdout { codec => rubydebug }
}

This works - the e-mails arrive in Teams as expected:

However, I can't now see the log entry in Kibana...

Is it possible that because the log entry has been processed by the email plugin, it's then not getting processed by the elasticsearch plugin?

[duncOps (Duncan Wraight)]

Little impatient bump (sorry - last day at my current organisation, would like to get some support with this if poss!).

[A_B]

Hi @duncOps,

I have used several outputs in parallel before. I suspect the if statement doens't share messages nicely...

I would either try putting the whole if block last (the other outputs would possibly get their "copy" before the if block) or add all outputs the each nested if statement.

[duncOps (Duncan Wraight)]

Thanks @A_B, I'll give that a go now.

[duncOps (Duncan Wraight)]

Bingo - putting the elasticsearch block first appears to have resolved the problem; I can now see the logs in both Teams and Kibana.

How bizarre...

Thanks again!

[Badger]

If you can provide a reproduceable configuration where events are not written to an output I would be very interested to see it.