Conditionals within filter using fields generated by cloudwatch logs plugin

[vacovsky (Joe Vacovsky)]

I'm having an issue getting my grok filter to apply to VPC logs, and hoping someone here can help.

filter {
    if "vpc-logs" in [cloudwatch_logs.log_group] {
        grok {
            patterns_dir => ["/etc/logstash/patterns/"]
            match => {"message" => "%{VPCFLOWLOG}"}
        }
    }
}

If I remove the "[cloudwatch_logs.log_group]" section and set it equal to a matching string (in this case: "aws-vpc-logs"), everything works fine, but the "cloudwatch_logs.log_group" field is generated by the input plugin I'm using. I'd like to do substring matching on this field to select the grok parser by name for each type of logs, using the AWS Cloudwatch Logs log group path as a determining factor. It seems as though the substring matching never passes the check.

Conversely, if I

I'd really like to be able to base my filter on a substring of the Cloudwatch Logs log_group name, for the sake of my sanity, and reducing complexity in the Logstash configurations I'm making.

[Badger]

Do you mean [cloudwatch][log_group] or do you really have a dot in the field name?

[vacovsky (Joe Vacovsky)]

Wow. Yes! That's exactly what I was doing wrong. Ditched the dot for your syntax and it started working.

For clarity: changed

if "vpc-logs" in [cloudwatch_logs.log_group] {

to

if "vpc-logs" in [cloudwatch_logs][log_group] {

Thanks very much!