Grokking the Linux authorization logs

[Feedy (F)]

Attempting to use a syslog pattern to parse ubuntu logs for building auth reports but having issues with syslog-filter.conf file. Below are the config files and error message:

Error message:

[2017-10-18T20:34:18,093][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-10-18T20:34:18,095][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-10-18T20:34:18,278][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, => at line 20, column 356 (byte 690) after filter {\n if [document_type] == "syslog" {\n grok {\n match => {"%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:system.auth.hostname} sshd(?:\\[%{POSINT:system.auth.pid}\\])?: %{DATA:system.auth.ssh.event} %{DATA:system.auth.ssh.method} for (invalid user )?%{DATA:system.auth.user} from %{IPORHOST:system.auth.ip} port %{NUMBER:system.auth.port} ssh2(: %{GREEDYDATA:system.auth.ssh.signature})?""}

Syslog-filter.conf(installed on logstash at /etc/logstash/conf.d/syslog-filter.conf):

filter {
if [document_type] == "syslog" {
grok {
match => {"%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:system.auth.hostname} sshd(?:\[%{POSINT:system.auth.pid}\])?: %{DATA:system.auth.ssh.event} %{DATA:system.auth.ssh.method} for (invalid user )?%{DATA:system.auth.user} from %{IPORHOST:system.auth.ip} port %{NUMBER:system.auth.port} ssh2(: %{GREEDYDATA:system.auth.ssh.signature})?"}
}
}
}

Filebeat.yml(from ubuntu host):

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

• input_type: log

  #Paths that should be crawled and fetched. Glob based paths.
  paths:

  • /var/syslog/home/ids/removed.ip/*.log
    exclude_files: [".gz$"]

  paths:

  • /var/log/auth.log
    document_type: syslog
    fields:
    client: removed
    fields_under_root: true

#----------------------------- Logstash output --------------------------------
output.logstash:
#The Logstash hosts
hosts: ["removed.ip.:5044"]
#loadbalance: true
bulk_max_size: 2048
#ssl.certificate_authorities: ["/etc/filebeat/logstash.crt"]
template.name: "filebeat"
template.path: "filebeat.template.json"
template.overwrite: false
index: removed

I've tried hunting down the line and column but I don't know what should be changed. Thanks in advance for any help.

[magnusbaeck (Magnus Bäck)]

Your grok filter isn't including the name of the field you want to parse. Compare your grok filter to the examples in the documentation and you'll spot your mistake.

[Feedy (F)]

Hi Magnus,
Thank you for your help, I located my issue, needed to added "message" after match in the syslog filter. But now I am getting a different error message:

[2017-10-19T20:56:59,938][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2017-10-19T20:57:01,980][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.
  Plugin: <LogStash::Inputs::Beats port=>5044, id=>"506b195a3ca1046d389dbb163621f089413fb78c-4", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_c28505b6-d915-4c17-a189-65d8ee6e271d", enable_metric=>true, charset=>"UTF-8">, host=>"0.0.0.0", ssl=>false, ssl_verify_mode=>"none", include_codec_tag=>true, ssl_handshake_timeout=>10000, congestion_threshold=>5, target_field_for_codec=>"message", tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60>
  Error: event executor terminated

Here is my updated filter config:

    filter {
     if [document_type] == "syslog" {
       grok {
           match => { "message" => "%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:system.auth.hostname} sshd(?:\\[%{POSINT:system.auth.pid}\\])?: %{DATA:system.auth.ssh.event} %{DATA:system.auth.ssh.method} for (invalid user )?%{DATA:system.auth.user} from %{IPORHOST:system.auth.ip} port %{NUMBER:system.auth.port} ssh2(: %{GREEDYDATA:system.auth.ssh.signature})?"}
      }
 }
}

[Feedy (F)]

Hi @magnusbaeck any help on the new error message I am seeing? Thanks in advance.