If-then json or grok

[randomuserid (Craig Chamberlain)]

So I am ingesting two json formatted logs plus most of the contents of /var/log including /var/log/nginx/. After some work I can parse the latter two categories of messages well with grok. I can send all of these to logstash via syslog-ng and tag the messages from the json files, using a “program” tag in this example.

Should if-then logic like this - or something else - work in a logstash filter? Can I use logic like this below, sending everything to grok if the program field is not a match? Can you mix json and grok in the same logstash pipeline? Do I need an else statement in there somewhere?

The logic is

1. If the program field matches suricata or goa, parse as json.
2. If not, send it to grok.
3. Ideally, stop on the first match, and parse each message using either json or grok but not both.

filter {

    if [program] == "suricata" {
        json {
                source => "message"
        }
    }

if [program] == "goa" {
json {
source => "message"
}
}

if [type] == "syslog" {
grok {many many grok statements}
}

[A_B]

Hi,

here's some examples of my if statements (all in the filter section)

  if ! [@metadata][index] {
    alter {
      add_field => {
        "[@metadata][index]" => "me"
      }
    }
  }

  if [@metadata][beat] {
    mutate {
      remove_field => [ "[host]" ]
    }
  }

  if "[fields][log_idx]" == "nginx-access" {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
      source => "clientip"
    }
  } 

Some of my logs are ingested as json some are system logs. Once the documents hit the filters they are all in Logstash format (what ever that is, probably json or something similar). I would assume your json logs is already parsed at the input so just use grok or any other filter the way you do for the other logs.

Hope that helps

-AB