Hi Everyone, I have multiline custom logs which I am processing as a single line by the filebeat multiline keyword. Now this includes \n at the end of each line. This however causes grok parse failure in my logstsash config file. Can someone help me on this. Here is how all of them look like:
MULTILINE LOG SAMPLE:
2/24/2017 12:05:59 AM :
Error thrown is:
Empty Queue
Request sent is:
cdii_hho_mx,201542318209,e034606f64a125c40b4b72592196a7a45cd
Response received is:
QSS RMS Holds Hashtable is empty
2/24/2017 12:05:59 AM :
Error thrown is:
Empty Queue
Request sent is:
sci_hho_it,2015423192624ftr6f64e5c40b4b45392196a7a45cd
Response received is:
QSS RMS Holds Hashtable is empty
Here is my filebeat.yml that puts it in a single line and sends it to redis:
-
input_type: log
Paths that should be crawled and fetched. Glob based paths.
paths:
- '\xcacad\HException*.txt'
document_type: batchlog_pgsexception
multiline.pattern: '\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} (AM|PM)\s*:'
multiline.negate: true
multiline.match: after
- '\xcacad\HException*.txt'
This is how my log looks like when it is sent to redis(it is treated as a single line but \n is added after each line end):
2/27/2017 7:44:13 AM : \nError thrown is:\nEmpty Queue\n*************************************************************************\nRequest sent is:\nhpi_hho_it,2015423192624,e06106f64e5c40b4b72592196a7a45cd\n*************************************************************************\nResponse received is:\nQSS RMS Holds Hashtable is empty\n*************************************************************************
and here is the config file in logstash:
%{DATE_US:logdate} %{TIME} ?%{WORD:AM_PM} : \n%{GREEDYDATA:Error}:\n%{GREEDYDATA:ErrorMessage}\n%{GREEDYDATA:lineseparator}\n%{GREEDYDATA:message}:\n{GREEDYDATA:requestdata}\n{GREEDYDATA:lineseparator1}\n%{GREEDYDATA:response}:\n%{GREEDYDATA:responsedata}\n%{GREEDYDATA:lineseparator2}
I tried some other ways to escape \n but that didn't work either. TIA