Logstash conf file will not work after filter section is added

Elastic Stack Logstash
1

I am trying to ingest logs from a checkpoint firewall. My logstash config file checks out and works as long as I don't have the filter section in it. However, I need the data filtered so can you guys help me figure out what is wrong with this filter. I pulled this from the Checkpoint website.

input {
tcp {
port => 2812
codec => plain
type => syslog
}
}
filter {
if [type] == "syslog" {
kv {
allow_duplicate_values => false
recursive => false
field_split => "|"
}

    mutate {
        # Fields beginning with underscore are not supported by ES/Kibana, so rename them.
         rename => { "__nsons" => "nsons" }
         rename => { "__p_dport" => "p_dport" }
         rename => { "__pos" => "pos" }
         # Not necessary, just a field name preference.
         rename => { "originsicname" => "sicname" }
         # Example of removing specific fields
         # remove_field => [ "connection_luuid", "loguid" ]
         # String substitution
         # Strip the O\=.*$ and the ^CN= in the field.
         gsub => [
              "sicname", "CN\\=", "",
              "sicname", ",O\\=.*$",""
         ]
    }
}

}
output {
elasticsearch {
hosts => ["https://localhost:9200"]
cacert => "C:\elastic\logstash-8.3.3-windows-x86_64\logstash-8.3.3\config\certs\http_ca.crt"
user => logstash_internal
password => ********
}
}

2

try

if ( "syslog" in [type]) {

3

I still get the same error.
[2022-08-09T14:30:59,784][ERROR][logstash.config.sourceloader] Could not fetch all the sources {:exception=>LogStash::ConfigLoadingError, :message=>"The following config files contains non-ascii characters but are not UTF-8 encoded ["c:/elastic/logstash-8.3.3-windows-x86_64/logstash-8.3.3/config/logstash2.conf"]", :backtrace=>["C:/elastic/logstash-8.3.3-windows-x86_64/logstash-8.3.3/logstash-core/lib/logstash/config/source/local.rb:99:in read'", "C:/elastic/logstash-8.3.3-windows-x86_64/logstash-8.3.3/logstash-core/lib/logstash/config/source/local.rb:110:in read'", "C:/elastic/logstash-8.3.3-windows-x86_64/logstash-8.3.3/logstash-core/lib/logstash/config/source/local.rb:206:in local_pipeline_configs'", "C:/elastic/logstash-8.3.3-windows-x86_64/logstash-8.3.3/logstash-core/lib/logstash/config/source/local.rb:177:in pipeline_configs'", "C:/elastic/logstash-8.3.3-windows-x86_64/logstash-8.3.3/logstash-core/lib/logstash/config/source_loader.rb:76:in block in fetch'", "org/jruby/RubyArray.java:2584:in collect'", "C:/elastic/logstash-8.3.3-windows-x86_64/logstash-8.3.3/logstash-core/lib/logstash/config/source_loader.rb:75:in fetch'", "C:/elastic/logstash-8.3.3-windows-x86_64/logstash-8.3.3/logstash-core/lib/logstash/runner.rb:389:in execute'", "C:/elastic/logstash-8.3.3-windows-x86_64/logstash-8.3.3/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:68:in run'", "C:/elastic/logstash-8.3.3-windows-x86_64/logstash-8.3.3/logstash-core/lib/logstash/runner.rb:283:in run'", "C:/elastic/logstash-8.3.3-windows-x86_64/logstash-8.3.3/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:133:in run'", "c:\\elastic\\logstash-8.3.3-windows-x86_64\\logstash-8.3.3\\lib\\bootstrap\\environment.rb:90:in '"]}
[2022-08-09T14:30:59,784][FATAL][logstash.runner ] The given configuration is invalid. Reason: Could not load the configuration file
[2022-08-09T14:30:59,800][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby.jar:?]
at c_3a_.elastic.logstash_minus_8_dot_3_dot_3_minus_windows_minus_x86_64.logstash_minus_8_dot_3_dot_3.lib.bootstrap.environment.(c:\elastic\logstash-8.3.3-windows-x86_64\logstash-8.3.3\lib\bootstrap\environment.rb:91) ~[?:?]

4

Without seeing exactly what is in that file (e.g. you upload the filters to a gist or other file sharing site) we cannot say what characters logstash is objecting to.

5

logstash2.conf is the config file that I initially pasted into my first post.

6

And as you can see some parts of it have been consumed as formatting, and in any case, non-ASCII characters may have been modified in the posting process.

In Powershell you could

Format-Hex -Path "c:/elastic/logstash-8.3.3-windows-x86_64/logstash-8.3.3/config/logstash2.conf"

and look for any ? in the rightmost column.

7

Ok, I ran the above command against my conf file and didn't find any question marks in the far right column. Very cool tool by the way. I am completely new at all of this and I am trying to get my initial build up and running against the checkpoint logs. Where do I go from here? I figure I will continue to get the following error from logstash if I don't deploy the filter that Checkpoint suggested.

[2022-08-09T10:45:07,789][ERROR][logstash.outputs.elasticsearch][main][83f3e8cc021b50a2ca04775d130dcc4754968715acc993cbd42bbdce83d20b67] Encountered a retryable error (will retry with exponential backoff) {:code=>403, :url=>"https://localhost:9200/_bulk", :content_length=>257005}

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.