Multiline log parsing from redis to logstash

Hi Everyone, I have multiline custom logs which I am processing as a single line by the filebeat multiline keyword. Now this includes \n at the end of each line. This however causes grok parse failure in my logstsash config file. Can someone help me on this. Here is how all of them look like:

MULTILINE LOG SAMPLE:

2/24/2017 12:05:59 AM :
Error thrown is:
Empty Queue


Request sent is:
cdii_hho_mx,201542318209,e034606f64a125c40b4b72592196a7a45cd


Response received is:
QSS RMS Holds Hashtable is empty


2/24/2017 12:05:59 AM :
Error thrown is:
Empty Queue


Request sent is:
sci_hho_it,2015423192624ftr6f64e5c40b4b45392196a7a45cd


Response received is:
QSS RMS Holds Hashtable is empty


Here is my filebeat.yml that puts it in a single line and sends it to redis:

  • input_type: log

    Paths that should be crawled and fetched. Glob based paths.

    paths:

    • '\xcacad\HException*.txt'
      document_type: batchlog_pgsexception
      multiline.pattern: '\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} (AM|PM)\s*:'
      multiline.negate: true
      multiline.match: after

This is how my log looks like when it is sent to redis(it is treated as a single line but \n is added after each line end):
2/27/2017 7:44:13 AM : \nError thrown is:\nEmpty Queue\n*************************************************************************\nRequest sent is:\nhpi_hho_it,2015423192624,e06106f64e5c40b4b72592196a7a45cd\n*************************************************************************\nResponse received is:\nQSS RMS Holds Hashtable is empty\n*************************************************************************

and here is the config file in logstash:

%{DATE_US:logdate} %{TIME} ?%{WORD:AM_PM} : \n%{GREEDYDATA:Error}:\n%{GREEDYDATA:ErrorMessage}\n%{GREEDYDATA:lineseparator}\n%{GREEDYDATA:message}:\n{GREEDYDATA:requestdata}\n{GREEDYDATA:lineseparator1}\n%{GREEDYDATA:response}:\n%{GREEDYDATA:responsedata}\n%{GREEDYDATA:lineseparator2}

I tried some other ways to escape \n but that didn't work either. TIA

What if you use \s, i.e. any whitespace character?

@magnusbaeck I had used \s in the yml file to denote whitespace, while I tried the same in logstash config but it had grok failure issues, I am guessing it has something to do with \n that was appended from filebeat. Can you kindly take a look at the grok filter pattern?

Also, how can I escape \\n or \nat the end of each line?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.