Hi,
I have some log files I want to parse in Filebeat and can't get the multiline.pattern options to work for my logs. All the sites I tested the regex and data at show it should match, but I get mixed up message entries in elasticsearch trying this.
Here is sample of 2 logs entries where I want each one to parse as a single message:
Date Time: 2017-05-03 15:11:25
Server Name: LEFFER-SB-02B
Agent ID: 14
User Name: LEFFER-SB-02B$
Error Number: -1
Error Description: 1The operation has timed outSystem.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.GetResponse()
at W6IntOutMsgLib.W6IntOutMsgGW.SendHTTPMessage(String& strMessage, String& strUrl, Boolean blnQueryString, Boolean blnAsync, String strSoapAction, String password, String userName, String domain, X509Certificate2 clientCertificate)
Error Source: W6IntUtilsLibGW.clsAgent.ProcessPendingMessages
Outgoing Message Key: 106991662
Incoming Message:
Date Time: 2017-05-03 15:13:09
Server Name: LEFFER-SB-02B
Agent ID: 14
User Name: LEFFER-SB-02B$
Error Number: -1
Error Description: 1The request was aborted: The operation has timed out.System.Net.WebException: The request was aborted: The operation has timed out.
at System.Net.HttpWebRequest.GetResponse()
at W6IntOutMsgLib.W6IntOutMsgGW.SendHTTPMessage(String& strMessage, String& strUrl, Boolean blnQueryString, Boolean blnAsync, String strSoapAction, String password, String userName, String domain, X509Certificate2 clientCertificate)
Error Source: W6IntUtilsLibGW.clsAgent.ProcessPendingMessages
Outgoing Message Key: 106991663
Incoming Message:
I couldn't figure out a way to get the blank line recognized as the break point between them. So I set up my filebeat.yml for this input file:
filebeat.inputs:
- type: log
enabled: true
tags:
["this_is_log3", "integration", "- type: log
enabled: true
tags:
["this_is_log3", "integration", "LEDCOR", "outgoing"]
paths:- C:\Temp\test_logs\W6IntLogOutgoing*
multiline.pattern: '/^Date Time:(?:.*)/'
multiline.negate: true
multiline.match: after
multiline.max_lines: 12000", "outgoing"]
paths: - C:\Temp\test_logs\W6IntLogOutgoing*
multiline.pattern: '/^Date Time:(?:.*)/'
multiline.negate: true
multiline.match: after
multiline.max_lines: 12000
- C:\Temp\test_logs\W6IntLogOutgoing*
When I look at it in Elasticsearch it just shows as one entry:
message Date Time: 2017-05-03 15:11:25 Server Name: LEFFER-SB-02B Agent ID: 14 User Name: LEFFER-SB-02B$ Error Number: -1 Error Description: <MessageResult Status="2"><Number>1</Number><Description>The operation has timed out</Description><Source>System.Net.WebException: The operation has timed out at System.Net.HttpWebRequest.GetResponse() at W6IntOutMsgLib.W6IntOutMsgGW.SendHTTPMessage(String& strMessage, String& strUrl, Boolean blnQueryString, Boolean blnAsync, String strSoapAction, String password, String userName, String domain, X509Certificate2 clientCertificate)</Source></MessageResult> Error Source: W6IntUtilsLibGW.clsAgent.ProcessPendingMessages Outgoing Message Key: 106991662 Incoming Message: Date Time: 2017-05-03 15:13:09 Server Name: LEFFER-SB-02B Agent ID: 14 User Name: LEFFER-SB-02B$ Error Number: -1 Error Description: <MessageResult Status="2"><Number>1</Number><Description>The request was aborted: The operation has timed out.</Description><Source>System.Net.WebException: The request was aborted: The operation has timed out. at System.Net.HttpWebRequest.GetResponse() at W6IntOutMsgLib.W6IntOutMsgGW.SendHTTPMessage(String& strMessage, String& strUrl, Boolean blnQueryString, Boolean blnAsync, String strSoapAction, String password, String userName, String domain, X509Certificate2 clientCertificate)</Source></MessageResult> Error Source: W6IntUtilsLibGW.clsAgent.ProcessPendingMessages Outgoing Message Key: 106991663 Incoming Message:
How can I make filebeat split the logs into 2 messages/events?
Thanks,
Fiona
filebeat version 6.3.2