Grok pattern is working if the "input" is a file, but not if the "input" is beats

Sorry about the long post. I have a multi-line CAS log I am shipping via filebeats to logstash running on a different server.

Here's a sample of the original multi-line log:

2020-06-21 00:24:00,833 INFO [org.apereo.inspektr.audit.support.Slf4jLoggingAuditTrailManager] - Audit trail record BEGIN
=============================================================
WHO: jane.doe
WHAT: [event=success,timestamp=Sun Jun 21 00:24:00 CDT 2020,source=RankedAuthenticationProviderWebflowEventResolver]
ACTION: AUTHENTICATION_EVENT_TRIGGERED
APPLICATION: CAS
WHEN: Sun Jun 21 00:24:00 CDT 2020
CLIENT IP ADDRESS: 10.10.10.10
SERVER IP ADDRESS: 20.20.20.20
=============================================================

I am using multiline option in my filebeat.yml to handle this. Here's my filebeat.yml sample file:

filebeat.inputs:

- type: log
  enabled: true
  paths:
      - /var/log/cas_sample.log

  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

output.logstash:
  hosts: ["30.30.30.30:5044"]

This results in the following single line log entry on the logstash side:

2020-06-21 00:24:00,833 INFO [org.apereo.inspektr.audit.support.Slf4jLoggingAuditTrailManager] - Audit trail record BEGIN\n=============================================================\nWHO: jane.doe\nWHAT: [event=success,timestamp=Sun Jun 21 00:24:00 CDT 2020,source=RankedAuthenticationProviderWebflowEventResolver]\nACTION: AUTHENTICATION_EVENT_TRIGGERED\nAPPLICATION: CAS\nWHEN: Sun Jun 21 00:24:00 CDT 2020\nCLIENT IP ADDRESS: 10.10.10.10\nSERVER IP ADDRESS: 20.20.20.20\n=============================================================

Here's my main logstash pipeline (it's the only one being used):

    input {
        beats {
            port => "5044"
        }
    }

    filter {
         grok {
            match => { "message" => "%{NOTSPACE:date} %{TIME:time} .*WHO: %{NOTSPACE:user_name}.*\\nWHAT.*ACTION: %{NOTSPACE:action}.*\\nAPPLICATION.*CLIENT IP ADDRESS: %{IP:source}.*SERVER IP ADDRESS: %{IP:destination}" }
         }
            remove_field => ["what", "message", "@version", "@timestamp", "path", "host"]
    }

    output {
        stdout { codec => rubydebug }
    }

But this doesn't do anything to the log. But if I change the pipeline input from beats to file and directly fed the single line log to logstash, then it works.

input {
    file {
        path => ["/tmp/cas_test.log"]
        sincedb_path => "/dev/null"
        start_position => "beginning"
    }
}

And the output would look like this:

         "action" => "AUTHENTICATION_EVENT_TRIGGERED",
         "source" => "10.10.10.10",
    "destination" => "20.20.20.20",
           "date" => "2020-06-21",
      "user_name" => "jane.doe",
           "time" => "00:24:00,833"

So what am I doing wrong? Can anyone tell me why beats input isn't working with this grok pattern?

Thanks!

what does the output look like when you use beats input?

Here's the output with when I use the beats input:

{
      "@version" => "1",
       "message" => "2020-06-21 00:24:00,833 INFO [org.apereo.inspektr.audit.support.Slf4jLoggingAuditTrailManager] - Audit trail record BEGIN\n=============================================================\nWHO: jane.doe\nWHAT: [event=success,timestamp=Sun Jun 21 00:24:00 CDT 2020,source=RankedAuthenticationProviderWebflowEventResolver]\nACTION: AUTHENTICATION_EVENT_TRIGGERED\nAPPLICATION: CAS\nWHEN: Sun Jun 21 00:24:00 CDT 2020\nCLIENT IP ADDRESS: 10.10.10.10\nSERVER IP ADDRESS: 20.20.20.20\n=============================================================",
         "agent" => {
                "type" => "filebeat",
        "ephemeral_id" => "SOME_ID",
                  "id" => "SOME_OTHER_ID",
            "hostname" => "SOME_HOST_NAME",
             "version" => "7.6.0"
    },
           "ecs" => {
        "version" => "1.4.0"
    },
    "@timestamp" => 2020-06-27T06:37:02.955Z,
           "log" => {
         "flags" => [
            [0] "multiline"
        ],
        "offset" => 1,
          "file" => {
            "path" => "/var/log/cas_sample.log"
        }
    },
          "tags" => [
        [0] "beats_input_codec_plain_applied",
        [1] "_grokparsefailure"
    ],
          "host" => {
        "name" => "SOME_HOST_NAME"
    },
         "input" => {
        "type" => "log"
    }
}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.