Where should I drop records? And how?

[mbuster (Matt Buster)]

Background:

• Sending Sonicwall syslog to Logstash in order to keep records on SSL VPN logins/logouts
• Sonicwall indices are accounting for a majority of my storage usage

Need:

• Drop Sonicwall records based on specific criteria

Detail:

• I'm using the generic Sonicwall filter that's been floating around for a while now and it works just fine for getting documents/events into Elasticsearch.
• In Elasticsearch, I'm able to filter events based on the 'msg' field. I'm only using two queries:
  • msg is SSL VPN zone remote user login allowed
  • msg is User logged out*
• I'd like to cut down on the Sonicwall logging data I'm storing and I figure I can drop records at logstash before they even coming into Elasticsearch
  • Thought I could tag records that have a specific 'msg' and then drop any records that don't have that tag. I can't seem to make this work...

Questions:

• Should I be dropping records in Logstash or Elasticsearch?
• Is Logstash breaking up the syslog event into its parsable chunks or is Elasticsearch?

I've attempted to add an 'if' statement to my logstash filter:

  if [msg] == \"SSL VPN zone remote user login allowed\" {
    mutate {
      add_tag => [ "valid" ]
    }
  }
  if "valid" not in [tags] {            
    drop { }
  }

But the above just crashed logstash... Even if I take out the second 'if' to do the drop, logstash doesn't like it.

Looking for guidance. Thanks!!

[Badger]

@mbuster

if [msg] == \"SSL VPN zone remote user login allowed\" {

Remove the backslashes before the double quotes.

Note that == is a string equality test. That would match the string

SSL VPN zone remote user login allowed

but not the string

SSL VPN zone remote user login allowed again

You could use a regexp match

if [msg] =~ "SSL VPN zone remote user login allowed" {

even through it is not a regular expression, or you could use a sub-string match

if "SSL VPN zone remote user login allowed" in [msg] {

[mbuster (Matt Buster)]

Badger -

Thank you so much for your reply and help!

It got me on the right track and now I'm successfully tagging records and dropping records that don't have the tag!

Thanks again!