Sorry about the long post. I have a multi-line CAS log I am shipping via filebeats to logstash running on a different server.
Here's a sample of the original multi-line log:
2020-06-21 00:24:00,833 INFO [org.apereo.inspektr.audit.support.Slf4jLoggingAuditTrailManager] - Audit trail record BEGIN
=============================================================
WHO: jane.doe
WHAT: [event=success,timestamp=Sun Jun 21 00:24:00 CDT 2020,source=RankedAuthenticationProviderWebflowEventResolver]
ACTION: AUTHENTICATION_EVENT_TRIGGERED
APPLICATION: CAS
WHEN: Sun Jun 21 00:24:00 CDT 2020
CLIENT IP ADDRESS: 10.10.10.10
SERVER IP ADDRESS: 20.20.20.20
=============================================================
I am using multiline
option in my filebeat.yml
to handle this. Here's my filebeat.yml
sample file:
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/cas_sample.log
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
output.logstash:
hosts: ["30.30.30.30:5044"]
This results in the following single line log entry on the logstash side:
2020-06-21 00:24:00,833 INFO [org.apereo.inspektr.audit.support.Slf4jLoggingAuditTrailManager] - Audit trail record BEGIN\n=============================================================\nWHO: jane.doe\nWHAT: [event=success,timestamp=Sun Jun 21 00:24:00 CDT 2020,source=RankedAuthenticationProviderWebflowEventResolver]\nACTION: AUTHENTICATION_EVENT_TRIGGERED\nAPPLICATION: CAS\nWHEN: Sun Jun 21 00:24:00 CDT 2020\nCLIENT IP ADDRESS: 10.10.10.10\nSERVER IP ADDRESS: 20.20.20.20\n=============================================================
Here's my main logstash pipeline (it's the only one being used):
input {
beats {
port => "5044"
}
}
filter {
grok {
match => { "message" => "%{NOTSPACE:date} %{TIME:time} .*WHO: %{NOTSPACE:user_name}.*\\nWHAT.*ACTION: %{NOTSPACE:action}.*\\nAPPLICATION.*CLIENT IP ADDRESS: %{IP:source}.*SERVER IP ADDRESS: %{IP:destination}" }
}
remove_field => ["what", "message", "@version", "@timestamp", "path", "host"]
}
output {
stdout { codec => rubydebug }
}
But this doesn't do anything to the log. But if I change the pipeline input
from beats
to file
and directly fed the single line log to logstash, then it works.
input {
file {
path => ["/tmp/cas_test.log"]
sincedb_path => "/dev/null"
start_position => "beginning"
}
}
And the output would look like this:
"action" => "AUTHENTICATION_EVENT_TRIGGERED",
"source" => "10.10.10.10",
"destination" => "20.20.20.20",
"date" => "2020-06-21",
"user_name" => "jane.doe",
"time" => "00:24:00,833"
So what am I doing wrong? Can anyone tell me why beats
input isn't working with this grok pattern?
Thanks!