Having issues with my PF sense configs

[JeMunos (Jesse Munos)]

I'm having issues with getting my PF sense logs to parse properly. The current result is that everything is dropped and I am uncertain why.
I am using the config files from this github: https://github.com/a3ilson/pfelk

I get this error:
[2019-10-06T00:16:31,031][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.4.0"}
[2019-10-06T00:16:32,625][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, ,, ] at line 45, column 28 (byte 844) after filter {\n if \"pf\" in [tags] {\n grok {\n match => [ \"message\" ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:incompile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:inblock in compile_sources'", "org/jruby/RubyArray.java:2584:in map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:incompile_sources'", "org/logstash/execution/AbstractPipelineExt.java:153:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:26:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326:in block in converge_state'"]}

I am uncertain what the issue is as I do get data into logstash and I can find no issues with the code that I am using. I thought this might be due to a version of logstash issue (i was on 6.x and I upgraded to 7.x to try to resolve) but the upgrade didn't fix anything.

I believe this is the file that is triggering the issue.

# 10-pf.conf
filter {
  if "pf" in [tags] {
    grok {
      match => [ "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}",
     "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" ]
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    mutate {
      rename => { "[message]" => "[event][original]"}
    }
  }
}

Any assistance would be appreciated

[Badger]

@JeMunos

match => [ "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp}

match expects a hash.

match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} ...

[JeMunos (Jesse Munos)]

So perhaps I don't understand the issue correctly (forgive me I'm a newb here). You said that match is expecting a hash, then gave me a modification of the match string using {} instead of . So I updated the code as follows to try to fix it:

# 10-pf.conf
filter {
  if "pf" in [tags] {
grok {
  match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}",
             "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
  add_field => [ "received_at", "%{@timestamp}" ]
  add_field => [ "received_from", "%{host}" ]
}
mutate {
  rename => { "[message]" => "[event][original]"}
}
}
}

This results in a slightly different error:

[2019-10-06T09:07:38,229][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.4.0"}
[2019-10-06T09:07:40,007][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 53, column 180 (byte 1097) after filter {\n  if \"pf\" in [tags] {\n    grok {\n      match => { \"message\" => \"%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\\[%{POSINT:syslog_pid}\\])?: %{GREEDYDATA:syslog_message}\"", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2584:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:153:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:26:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326:in `block in converge_state'"]}
[2019-10-06T09:07:40,327][INFO ][logstash.runner          ] Logstash shut down.

Perhaps you were suggesting that I have to change the datatype being passed into match, or use a different method all together?

[Badger]

@JeMunos

match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}", 
           "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" } add_field => [ "received_at", "%{@timestamp}" ]

The key value pairs in a hash are not comma separated. So you can get past that error message by using

match => { "message" => "some pattern"
                    "message" => "some other pattern" }

However, if you want to match [message] against multiple patterns then you should use

match => { "message" => [ "some pattern", "some other pattern" ] }

[JeMunos (Jesse Munos)]

Thank you. Changing it to use the multiple pattern approach resolved my issue.