Assistance with my Logstash filter

Elastic Stack Logstash
1

Hello,

I am trying to use a variation of the filter provided here - https://github.com/bromiley/olaf/blob/master/logstash/o365.config but I keep getting the following error,

[FATAL][logstash.runner ] The given configuration is invalid. Reason: Expected one of [ \t\r\n], "#", "=>" at line 17, column 12 (byte 616) after filter {

Below is the filter config I am using. Any suggestions on how to correct this? Thanks!

filter {
  if [log][file][path] == "/var/log/audit.log"
    {
    kv {
     include_keys => [ "VlogRecNo", "VigilRecNo", "Pid", "TimeStamp", "Type", "Event", "LinuxPath", "netAddr_IPv4", "Dn", "UserDn", "PATH Type", "Comm" ]
       }
    if [Comm] == "smdrd"
     {
        drop { }
     }
    }

  if ([type] == "o365_csv") {
      csv {
        columns => ["PSComputerName", "RunspaceId", "PSShowComputerName", "RecordType", "CreationDate", "UserIds", "Operations", "AuditData", "ResultIndex", "ResultCount", "Identity", "IsValid", "ObjectState"]
        skip_header => "true"
        if ([message =~ /^#/) {
          drop {}
        }
      }
      date {
        match => [ "CreationDate", "ISO8601" ]
      }
      json {
        source => "AuditData"
      }
  }
}
2

Realized it was missing a square bracket after message (see below). Added the missing bracket but it's still throwing the same error

if ([message] =~ /^#/) {
3

Could be missing the } to close a previous input, output, or filter section.

What do the first 18 lines of the configuration look like?

4

Thanks for the reply @Badger! That's the whole filter file posted above. I copied and pasted the contents from above into http://www.yamllint.com/ and it says it's valid. The error has me baffled.

5

What other files are part of the configuration?

6

The input/output section is another file in that same folder. Password has been removed on post.

input {
  tcp {
    port => 5544
  }

  beats {
    port => 5044
    ssl => false
  }

}

output {
#  if [@metadata][beat] == "filebeat" and [input][type] == "netflow" {
#    elasticsearch {
#      hosts => ["elastic1", "elastic2", "elastic3"]
#      index => "netflow-%{[@metadata][version]}"
#      user => "service-account"
#      password => ""
#    }
#  }
  if [@metadata][beat] == "filebeat" {
    elasticsearch {
      hosts => ["elastic1", "elastic2", "elastic3"]
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      user => "service-account"
      password => ""
      ssl => "true"
    }
  }
  if [@metadata][beat] == "winlogbeat" {
    elasticsearch {
      hosts => ["elastic1", "elastic2", "elastic3"]
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      user => "service-account"
      password => ""
      ssl => "true"
    }
  }
#  else {
#    elasticsearch {
#      hosts => ["elastic1", "elastic2", "elastic3"]
#      index => "syslog-%{+YYYY.MM.dd}"
#      user => "service-account"
#      password => ""
#    }
#  }
}
7

Try running with --log.level debug --config.debug and see what it is trying to load. Show us the first 18 lines of the configuration that it loads, you can redact anything you need to, but do not add or delete any lines.

8

Hope this helps

[2020-10-08T11:46:21,890][DEBUG][org.logstash.config.ir.PipelineConfig] -------- Logstash Config ---------
[2020-10-08T11:46:21,896][DEBUG][org.logstash.config.ir.PipelineConfig] Config from source, source: LogStash::Config::Source::Local, pipeline_id:: main
[2020-10-08T11:46:21,898][DEBUG][org.logstash.config.ir.PipelineConfig] Config string, protocol: file, id: /etc/logstash/conf.d/filter.conf
[2020-10-08T11:46:21,899][DEBUG][org.logstash.config.ir.PipelineConfig]

filter {
  if [log][file][path] == "/var/log/audit.log"
    {
    kv {
     include_keys => [ "VlogRecNo", "VigilRecNo", "Pid", "TimeStamp", "Type", "Event", "LinuxPath", "netAddr_IPv4", "Dn", "UserDn", "PATH Type", "Comm" ]
       }
    if [Comm] == "smdrd"
     {
        drop { }
     }
    }

  if ([type] == "o365_csv") {
      csv {
        columns => ["PSComputerName", "RunspaceId", "PSShowComputerName", "RecordType", "CreationDate", "UserIds", "Operations", "AuditData", "ResultIndex", "ResultCount", "Identity" "IsValid", "ObjectState"]
        skip_header => "true"
        if ([message] =~ /^#/) {
          drop { }
        }
      }
      date {
        match => [ "CreationDate", "ISO8601" ]
      }
      json {
        source => "AuditData"
      }
  }
}
9 
    if ([message] =~ /^#/) {
      drop { }
    }

You cannot put that inside the csv {} filter.

10

Thanks, that was it! Removed that part and it's now working.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.