Configuration error - Transcribing auditd ingest pipeline to logstash config

Elastic Stack Logstash
1

I'm attempting to transcribe the auditd ingest pipeline from the filebeat module to a logstash config, and I'm hitting the following config syntax error. Any idea what I'm missing?

   [2017-05-08T18:05:19,699][FATAL][logstash.runner          ] The given configuration is invalid. Reason: Expected one of #, {, } at line 20, column 67 (byte 893) after filter {
       grok {
          match => { "message" => [
                   "%{AUDIT_PREFIX} %{AUDIT_KEY_VALUES:auditd.log.kv} old auid=%{NUMBER:auditd.log.old_auid} new auid=%{NUMBER:auditd.log.new_auid} old ses=%{NUMBER:auditd.log.old_ses} new ses=%{NUMBER:auditd.log.new_ses}",
                   "%{AUDIT_PREFIX} %{AUDIT_KEY_VALUES:auditd.log.kv} msg=['\"](%{DATA:auditd.log.msg}\\s+)?%{AUDIT_KEY_VALUES:auditd.log.sub_kv}['\"]",
                   "%{AUDIT_PREFIX} %{AUDIT_KEY_VALUES:auditd.log.kv}",
                   "%{AUDIT_PREFIX}",
                   "%{AUDIT_TYPE} %{AUDIT_KEY_VALUES:auditd.log.kv}"
                   ]
                   }
          pattern_definitions => {
            "AUDIT_TYPE" => "^type=%{NOTSPACE:auditd.log.record_type}"
2

Also line 20, column 67 is the end of this line:
"AUDIT_TYPE" => "^type=%{NOTSPACE:auditd.log.record_type}"

3

Full config:

input {
  beats {
    # The port to listen on for filebeat connections.
    port => 5044
    # The IP address to listen for filebeat connections.
    host => "0.0.0.0"
  }
}
filter {
   grok {
      match => { "message" => [
               "%{AUDIT_PREFIX} %{AUDIT_KEY_VALUES:auditd.log.kv} old auid=%{NUMBER:auditd.log.old_auid} new auid=%{NUMBER:auditd.log.new_auid} old ses=%{NUMBER:auditd.log.old_ses} new ses=%{NUMBER:auditd.log.new_ses}",
               "%{AUDIT_PREFIX} %{AUDIT_KEY_VALUES:auditd.log.kv} msg=['\"](%{DATA:auditd.log.msg}\\s+)?%{AUDIT_KEY_VALUES:auditd.log.sub_kv}['\"]",
               "%{AUDIT_PREFIX} %{AUDIT_KEY_VALUES:auditd.log.kv}",
               "%{AUDIT_PREFIX}",
               "%{AUDIT_TYPE} %{AUDIT_KEY_VALUES:auditd.log.kv}"
               ]
               }
      pattern_definitions => {
        "AUDIT_TYPE" => "^type=%{NOTSPACE:auditd.log.record_type}",
        "AUDIT_PREFIX" => "%{AUDIT_TYPE} msg=audit\\(%{NUMBER:auditd.log.epoch}:%{NUMBER:auditd.log.sequence}\\):(%{DATA})?",
        "AUDIT_KEY_VALUES" => "%{WORD}=%{GREEDYDATA}"
      }
      remove_field => "message"
   }
   kv {
      source => "auditd.log.kv",
      field_split => "\\s+",
      value_split => "=",
      target => "auditd.log",
      remove_field => "auditd.log.kv"
   }
   kv {
      source => "auditd.log.sub_kv",
      field_split => "\\s+",
      value_split => "=",
      target => "auditd.log",
      remove_field => "auditd.log.sub_kv"
   }
   date {
      match => [ "auditd.log.epoch", "UNIX" ],
      target_field => "@timestamp",
      remove_field => "auditd.log.epoch"
   }
   geoip {
      source => "auditd.log.addr"
      target => "auditd.log.geoip"
   }
}
output {
  elasticsearch {
    hosts => localhost
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}
4

Hash pairs aren't separated by commas, i.e. it should be

  pattern_definitions => {
    "AUDIT_TYPE" => "^type=%{NOTSPACE:auditd.log.record_type}"
    "AUDIT_PREFIX" => "%{AUDIT_TYPE} msg=audit\\(%{NUMBER:auditd.log.epoch}:%{NUMBER:auditd.log.sequence}\\):(%{DATA})?"
    "AUDIT_KEY_VALUES" => "%{WORD}=%{GREEDYDATA}"
  }

not

  pattern_definitions => {
    "AUDIT_TYPE" => "^type=%{NOTSPACE:auditd.log.record_type}",
    "AUDIT_PREFIX" => "%{AUDIT_TYPE} msg=audit\\(%{NUMBER:auditd.log.epoch}:%{NUMBER:auditd.log.sequence}\\):(%{DATA})?",
    "AUDIT_KEY_VALUES" => "%{WORD}=%{GREEDYDATA}"
  }
2 Likes
5

Thanks! That solved it.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.