Grokking the Linux authorization logs

Attempting to use a syslog pattern to parse ubuntu logs for building auth reports but having issues with syslog-filter.conf file. Below are the config files and error message:

Error message:

[2017-10-18T20:34:18,093][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-10-18T20:34:18,095][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-10-18T20:34:18,278][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, => at line 20, column 356 (byte 690) after filter {\n if [document_type] == "syslog" {\n grok {\n match => {"%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:system.auth.hostname} sshd(?:\\[%{POSINT:system.auth.pid}\\])?: %{DATA:system.auth.ssh.event} %{DATA:system.auth.ssh.method} for (invalid user )?%{DATA:system.auth.user} from %{IPORHOST:system.auth.ip} port %{NUMBER:system.auth.port} ssh2(: %{GREEDYDATA:system.auth.ssh.signature})?""}

Syslog-filter.conf(installed on logstash at /etc/logstash/conf.d/syslog-filter.conf):

filter {
if [document_type] == "syslog" {
grok {
match => {"%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:system.auth.hostname} sshd(?:\[%{POSINT:system.auth.pid}\])?: %{DATA:system.auth.ssh.event} %{DATA:system.auth.ssh.method} for (invalid user )?%{DATA:system.auth.user} from %{IPORHOST:system.auth.ip} port %{NUMBER:system.auth.port} ssh2(: %{GREEDYDATA:system.auth.ssh.signature})?"}
}
}
}

Filebeat.yml(from ubuntu host):

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

  • input_type: log

    #Paths that should be crawled and fetched. Glob based paths.
    paths:

    • /var/syslog/home/ids/removed.ip/*.log
      exclude_files: [".gz$"]

    paths:

    • /var/log/auth.log
      document_type: syslog
      fields:
      client: removed
      fields_under_root: true

#----------------------------- Logstash output --------------------------------
output.logstash:
#The Logstash hosts
hosts: ["removed.ip.:5044"]
#loadbalance: true
bulk_max_size: 2048
#ssl.certificate_authorities: ["/etc/filebeat/logstash.crt"]
template.name: "filebeat"
template.path: "filebeat.template.json"
template.overwrite: false
index: removed

I've tried hunting down the line and column but I don't know what should be changed. Thanks in advance for any help.

Your grok filter isn't including the name of the field you want to parse. Compare your grok filter to the examples in the documentation and you'll spot your mistake.

Hi Magnus,
Thank you for your help, I located my issue, needed to added "message" after match in the syslog filter. But now I am getting a different error message:

[2017-10-19T20:56:59,938][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2017-10-19T20:57:01,980][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.
  Plugin: <LogStash::Inputs::Beats port=>5044, id=>"506b195a3ca1046d389dbb163621f089413fb78c-4", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_c28505b6-d915-4c17-a189-65d8ee6e271d", enable_metric=>true, charset=>"UTF-8">, host=>"0.0.0.0", ssl=>false, ssl_verify_mode=>"none", include_codec_tag=>true, ssl_handshake_timeout=>10000, congestion_threshold=>5, target_field_for_codec=>"message", tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60>
  Error: event executor terminated

Here is my updated filter config:

    filter {
     if [document_type] == "syslog" {
       grok {
           match => { "message" => "%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:system.auth.hostname} sshd(?:\\[%{POSINT:system.auth.pid}\\])?: %{DATA:system.auth.ssh.event} %{DATA:system.auth.ssh.method} for (invalid user )?%{DATA:system.auth.user} from %{IPORHOST:system.auth.ip} port %{NUMBER:system.auth.port} ssh2(: %{GREEDYDATA:system.auth.ssh.signature})?"}
      }
 }
}

Hi @magnusbaeck any help on the new error message I am seeing? Thanks in advance.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.