Grok pattern is working if the "input" is a file, but not if the "input" is beats

[Aj_t]

Sorry about the long post. I have a multi-line CAS log I am shipping via filebeats to logstash running on a different server.

Here's a sample of the original multi-line log:

2020-06-21 00:24:00,833 INFO [org.apereo.inspektr.audit.support.Slf4jLoggingAuditTrailManager] - Audit trail record BEGIN
=============================================================
WHO: jane.doe
WHAT: [event=success,timestamp=Sun Jun 21 00:24:00 CDT 2020,source=RankedAuthenticationProviderWebflowEventResolver]
ACTION: AUTHENTICATION_EVENT_TRIGGERED
APPLICATION: CAS
WHEN: Sun Jun 21 00:24:00 CDT 2020
CLIENT IP ADDRESS: 10.10.10.10
SERVER IP ADDRESS: 20.20.20.20
=============================================================

I am using multiline option in my filebeat.yml to handle this. Here's my filebeat.yml sample file:

filebeat.inputs:

- type: log
  enabled: true
  paths:
      - /var/log/cas_sample.log

  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

output.logstash:
  hosts: ["30.30.30.30:5044"]

This results in the following single line log entry on the logstash side:

2020-06-21 00:24:00,833 INFO [org.apereo.inspektr.audit.support.Slf4jLoggingAuditTrailManager] - Audit trail record BEGIN\n=============================================================\nWHO: jane.doe\nWHAT: [event=success,timestamp=Sun Jun 21 00:24:00 CDT 2020,source=RankedAuthenticationProviderWebflowEventResolver]\nACTION: AUTHENTICATION_EVENT_TRIGGERED\nAPPLICATION: CAS\nWHEN: Sun Jun 21 00:24:00 CDT 2020\nCLIENT IP ADDRESS: 10.10.10.10\nSERVER IP ADDRESS: 20.20.20.20\n=============================================================

Here's my main logstash pipeline (it's the only one being used):

    input {
        beats {
            port => "5044"
        }
    }

    filter {
         grok {
            match => { "message" => "%{NOTSPACE:date} %{TIME:time} .*WHO: %{NOTSPACE:user_name}.*\\nWHAT.*ACTION: %{NOTSPACE:action}.*\\nAPPLICATION.*CLIENT IP ADDRESS: %{IP:source}.*SERVER IP ADDRESS: %{IP:destination}" }
         }
            remove_field => ["what", "message", "@version", "@timestamp", "path", "host"]
    }

    output {
        stdout { codec => rubydebug }
    }

But this doesn't do anything to the log. But if I change the pipeline input from beats to file and directly fed the single line log to logstash, then it works.

input {
    file {
        path => ["/tmp/cas_test.log"]
        sincedb_path => "/dev/null"
        start_position => "beginning"
    }
}

And the output would look like this:

         "action" => "AUTHENTICATION_EVENT_TRIGGERED",
         "source" => "10.10.10.10",
    "destination" => "20.20.20.20",
           "date" => "2020-06-21",
      "user_name" => "jane.doe",
           "time" => "00:24:00,833"

So what am I doing wrong? Can anyone tell me why beats input isn't working with this grok pattern?

Thanks!

[ptamba]

what does the output look like when you use beats input?

[Aj_t]

Here's the output with when I use the beats input:

{
      "@version" => "1",
       "message" => "2020-06-21 00:24:00,833 INFO [org.apereo.inspektr.audit.support.Slf4jLoggingAuditTrailManager] - Audit trail record BEGIN\n=============================================================\nWHO: jane.doe\nWHAT: [event=success,timestamp=Sun Jun 21 00:24:00 CDT 2020,source=RankedAuthenticationProviderWebflowEventResolver]\nACTION: AUTHENTICATION_EVENT_TRIGGERED\nAPPLICATION: CAS\nWHEN: Sun Jun 21 00:24:00 CDT 2020\nCLIENT IP ADDRESS: 10.10.10.10\nSERVER IP ADDRESS: 20.20.20.20\n=============================================================",
         "agent" => {
                "type" => "filebeat",
        "ephemeral_id" => "SOME_ID",
                  "id" => "SOME_OTHER_ID",
            "hostname" => "SOME_HOST_NAME",
             "version" => "7.6.0"
    },
           "ecs" => {
        "version" => "1.4.0"
    },
    "@timestamp" => 2020-06-27T06:37:02.955Z,
           "log" => {
         "flags" => [
            [0] "multiline"
        ],
        "offset" => 1,
          "file" => {
            "path" => "/var/log/cas_sample.log"
        }
    },
          "tags" => [
        [0] "beats_input_codec_plain_applied",
        [1] "_grokparsefailure"
    ],
          "host" => {
        "name" => "SOME_HOST_NAME"
    },
         "input" => {
        "type" => "log"
    }
}