Logstash – Both elasticsearch and email outputs

Elastic Stack Logstash
1

Hello,

Please can some provide an example config that provides two
outputs of the same input (syslog)?

I’m wanting to pass syslogs both to elasticserarch and
email.

The below is my current output conf, the e-mail section
works but I’m struggling with including both the elastic one and email:

output {
#  elasticsearch { host => localhost }
#  stdout { codec => rubydebug }
#}
email {
        to => "mysmtp"
        from => "ELK-Alert-Notifs@domain"
        via => "smtp"
        subject => "ELK Syslog Alerting Concerning Host %{host}."
        body => "This syslog has an interesting string and thus picked up on host (%{host}). Via this message: %{message}"
}
}

Note I’ve deliberately hashed out the entries above the
email plugin.

2

The email output needs to be inside the output block.

output {
  elasticsearch {
    ...
  }
  email {
    ..
  }
}
3

I believe I’ve done this as you advised:

    output {
      elasticsearch { host => localhost }
      stdout { codec => rubydebug }
    }
    email {
            to => "mysmtp"
            from => "ELK-Alert-Notifs@mydomain"
            via => "smtp"
            subject => "ELK Syslog Alerting Concerning Host %{host}."
            body => "This syslog has an interesting string and thus picked up on host (%{host}). Via this message: %{message}"
    }
 }
4

No, you have an extra closing brace on the line right before "email". That brace closes the output block.

5

Thanks Magnus that worked.

Just one last question, we'll be wanting to use maybe the IF module to check for strings/REGEX in the syslogs and e-mail them out, everything would be syslogged but only this string/regex would be e-mailed.

I'm guessing if "string/regex" in [message] { would be the syntax but where exactly would it go in my config please?

6

Yes, you'll want to use a conditional block here. See the documentation for details and examples. Your conditional should surround the email output.

if ... {
  email {
    ...
  }
}
7

I have this config to send emails on if Apache exists in the
message field, but it sends e-mails out regardless. I’m guessing the IF
statement is correct but there is no closing such as IF not exist and the email
plugin is still being used?

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
if [message] in "Apache" {
email {
        to => "mysmtp"
        from => "ELK-Alert-Notifs@mydomain"
        via => "smtp"
        subject => "ELK Syslog Alerting Concerning Host %{host}."
        body => "This syslog has an interesting string and thus picked up on host (%{host}). Via this message: %{message}"
   }
 }
}
8

Your conditional is backwards. This is what you should use:

if "Apache" in [message] {

It's surprising that you get any email. I'd imagine that your backwards conditional would never match.

9

I still seem to get all the e-mails out of both outputs
rather than one only with a string, this is what the output looks like:

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
if "Apache" in [message] {
email {
        to => "ben.lavender@virtualdcs.co.uk"
        from => "ELK-Alert-Notifs@Nimbox.co.uk"
        via => "smtp"
        subject => "ELK Syslog Alerting Concerning Host %{host}."
        body => "This syslog has an interesting string and thus picked up on host (%{host}). Via this message: %{message}"
   }
 }
}

Maybe it's an issue with the grok coding in my filter conf:

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}
10

Are you running Logstash as a daemon or by hand from a shell? In the former case, do you have any additional files in /etc/logstash/conf.d? Logstash will read (almost) all files in that directory.

11

Good question about other files in that directory, I was
aware logstash will read files any .conf in that dir, having said that I've
found a file named ~? and it contained the below:

output {
#  elasticsearch { host => localhost }
#  stdout { codec => rubydebug }
#}
email {
        to => "mysmtp"
        from => "ELK-Alert-Notifs@mydomain"
        via => "smtp"
        subject => "ELK Syslog Alerting Concerning Host %{host}."
        body => "This syslog has an interesting string and thus picked up on host (%{host}). Via this message: %{message}"
}
}

So obvisuly that was the problem and it was configured to
bang it all out via e-mail.

I have been backing up the .confs to ~/ and must have at some
point just missed the / off and it’s placed it back in the working directly.

I’m testing this now (so far so good) and will
update you shortly Magnus, thanks so far btw.

12

If the file really is named ~? (tilde questionmark) you probably accidentally pressed the shift key, causing the slash you intended to become a question mark.

Configuration files whose name ends with a tilde are ignored (below) but this doesn't cover files with an extra question mark at the end.

13

Thanks Magnus I’ll take note of that. I’ve got an issue that
has now developmed on one of our single-node clusters, strange as this exact
output conf resulting in correct opersation during testing but as soon as I
loaded it onto the other cluster (all /etc/logstash/conf.d/*.conf files exactly
the same the server doesn’t generate an e-mail.

I’ve even ran a pcap using TCPdump and no smtp data is even
generated…

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
if "Apache" in [message] {
email {
        to => "mysmtp"
        from => "Syslog-Alerts@domain.com"
        via => "smtp"
        subject => "ELK Syslog Alerting Concerning Host %{host}."
        body => "This syslog has an interesting string and thus picked up on host (%{host}). Via this message: %{message}"
   }
} else if "[mpm_winnt:notice]" in [message] {
email {
        to => "mysmtp"
        from => "Syslog-Alerts@domain.com"
        via => "smtp"
        subject => "MPM Notice from Web Host %{host}."
        body => "%{message}"
  }
} else if "InstallType: upgrade" in [message] {
email {
        to => "mysmtp"
        from => "Syslog-Alerts@domain.com"
        via => "smtp"
        subject => "An Upgrade Notice on Host: %{host}."
        body => "%{message}"
  }
} else if "[BST]" in [message] {
email {
        to => "mysmtp"
        from => "Syslog-Alerts@domain.com"
        via => "smtp"
        subject => "PostgreSQL System Log from %{host}."
        body => "I would suggest looking into BST logs since we turned off PG debugging, here is the message field: %{message}"
  }
 }
}

I’ve edited out the smtps btw

14

did you get this figured out? if so, can you post up the config files?

I am also working on emailing the log if a certain condition is met.

15

I can if you still need it?

16

please do so, thanks.

17

Couple of people have contacted me for the code, a simple one is below:

output {

elasticsearch {
hosts => localhost
index => "syslogs-%{+YYYY.MM.dd}" }
stdout { codec => rubydebug }
if "string" in [message] {
email {
to => "smtp_address"
from => "smtp_address"
via => "smtp"
subject => "Subject from %{host}."
body => "Body"
}
}
}