Grok filter failing multiline logs from filebeat

[mladen (Mladen Stankovic)]

Hello,

I have problem with parsing multiline log messages from Filebeat to Logstash using grok filter. This is my configuration of filebeat.yml:

filebeat.inputs:

    - type: log
      paths:
        - "/home/mladen/Desktop/qmraz2.log"
      fields:
        qmraz2: true
      fields_under_root: true

      ### Multiline options

      # Mutiline can be used for log messages spanning multiple lines. This is common
      # for Java Stack Traces or C-Line Continuation

      # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
      multiline.pattern: '^AMQ8409'

      # Defines if the pattern set under pattern should be negated or not. Default is false.
      multiline.negate: true

      # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
      # that was (not) matched before or after or as long as a pattern is not matched based on negate.
      # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
      multiline.match: after 

      multiline.flush_pattern: 5

This is sample of my log file:

AMQ8409: Display Queue details.
   QUEUE(SYSTEM.CHLAUTH.DATA.QUEUE)        TYPE(QLOCAL)
   CURDEPTH(3)
AMQ8409: Display Queue details.
   QUEUE(SYSTEM.CLUSTER.REPOSITORY.QUEUE)
   TYPE(QLOCAL)                            CURDEPTH(2)
AMQ8409: Display Queue details.
   QUEUE(SYSTEM.DURABLE.SUBSCRIBER.QUEUE)
   TYPE(QLOCAL)                            CURDEPTH(1)

Filebeat combine lines from log file ok, but from some reason all lines go to grok failures file instead of Elastic. This is my Logstash pipeline:

input {
    beats {
        port => "5044"
    }
}

filter {
    if [qmraz2] {
        grok {
            match => { "message" => "^AMQ8409: Display Queue details.\\n%{SPACE}QUEUE\(%{NOTSPACE:queue_name}\)(\\n)?%{SPACE}TYPE\(QLOCAL\)(\\n)?%{SPACE}CURDEPTH\(%{NUMBER:curdepth:int}\)(\\n)?%{SPACE}$" }
        }
    }    
}

output {
    if [qmraz2] {
        if "_grokparsefailure" in [tags] {
            # write events that didn't match to a file
            file { "path" => "/home/mladen/Desktop/grok_qmraz2_failures_kaiibraz.txt" }
        }

        else {
            file { "path" => "/home/mladen/Desktop/grok_qmraz2_sucess_kaiibraz.txt" }
        }        

    }    
}

This is sample message from grok failure file:

message":"AMQ8409: Display Queue details.\n QUEUE(SYSTEM.CHLAUTH.DATA.QUEUE) TYPE(QLOCAL)\n CURDEPTH(3) "

On production I have Filebeat version 6.2.2 and Logstash version 6.3.1. To be sure that I don’t have any other filter that collide with this one I have created isolated environment and reproduced same problem on Filebeat version 6.3.1 and Logstash 6.3.1 version.

First, I was thinking that the problem is grok filter but after testing every message line from grok failure file on test pipeline with same grok filter I release that I don't have a clue what is causing the problem

BR,
Mladen

[mladen (Mladen Stankovic)]

Hi,

I found out what was the problem. Problem was nonexistent \n (new line) in my grok filter when output is processed through logstash. My assumption was that this symbol should exist here as this is the case when output is saved in the file.

BR,
Mladen