Multiline filter not working as expected

Elastic Stack Beats
1

Hi Elastic/Filebeat Team,

I am trying to configure filebeat along with logstash and Elasticsearch but not getting expected result in Kibana dashboard as logs are getting split into multiple line please find sample logs below:

DATA FLOW : Filebeat >> Logstash >> Elasticsearch >> Kibana

Logs Structure:
Hi Elastic Team,

I am trying to configure filebeat along with logstash and Elasticsearch but not getting expected result in Kibana dashboard as logs are getting split into multiple line please find sample logs and filebeat config below:

Logs Structure:
2019 Jan 20 23:53:40:360 GMT +0530 BW.xyz-xyz Info [BW-User] - Job-7851213-2 [Logger/LogToLocal.process/Log]: Error in: Services/xyz/Status02_02.process/Status02>Processes/time/Status02.process/qwerty.xyz>Processes/References/qwerty/qwerty.xyz.process/Invoke qwerty par Status check with request payload: <?xml version="1.0" encoding="UTF-8"?> <ns0:Status02Request xmlns:ns0="http://www.abc.com/IN/abc"><MessageHeader><head:MessageHeader xmlns:head="http://www.abc.com/IT/EAG/Common/Header_01" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns="http://www.abc.com/IT/EAG/Common/Header_01" xmlns:ns0="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:v1="http://www.abc.com/IN/new/mysee/yiu/v1" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><head:From><1234</head:ConversationId><head:MessageId>1234</head:MessageId><head:BusinessObjectId>Acc</head:BusinessObjectId><head:Timestamp>2019-01-20T23:53:40+05:30</head:Timestamp></head:HeaderFields></head:MessageHeader></MessageHeader><MessageBody><ns0:Status02Request xmlns:ns0="http://www.abc.com/IN/new/mysee/Status/v2" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><TxnId>AVC1234</TxnId></ns0:Status02Request></MessageBody></ns0:Status02Request> and error message: An IOException was thrown while trying to execute the Http method and stack trace: Job-7851213 Error in [Processes/References/qwerty/qwerty.xyz.process/Invoke qwerty par Status check] An IOException was thrown while trying to execute the Http method at com.myapp.plugin.share.http.client.JakartaHttpTransportDriver$RequestExecutor.run(Unknown Source) at com.myapp.pe.util.ThreadPool$ThreadPoolThread.run(Unknown Source) caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at java.net.Socket.connect(Socket.java:538) at java.net.Socket.<init>(Socket.java:434) at java.net.Socket.<init>(Socket.java:286) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:79) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:121) at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:706) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:386) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324) at com.myapp.plugin.share.http.client.JakartaHttpTransportDriver$RequestExecutor.run(Unknown Source) at com.myapp.pe.util.ThreadPool$ThreadPoolThread.run(Unknown Source)
2019 Jan 20 23:54:40:360 GMT +0530 BW.xyz-xyz Info [BW-User] - Job-7851213-2 [Logger/LogToLocal.process/Log]: Error in: Services/xyz/Status02_02.process/Status02>Processes/time/Status02.process/qwerty.xyz>Processes/References/qwerty/qwerty.xyz.process/Invoke qwerty par Status check with request payload: &lt xmlns:ns0="http://www.abc.com/IN/abc"><MessageHeader><head:MessageHeader xmlns:head="http://www.abc.com/IT/EAG/Common/Header_01" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns="http://www.abc.com/IT/EAG/Common/Header_01" xmlns:ns0="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:v1="http://www.abc.com/IN/" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><head:From><head:Id></head:Id></head:From><head:HeaderFields><head:ConversationId>19012</head:ConversationId><head:MessageId>190</head;/ns0:></MessageBody></ns0:est> and error message: An IOException was thrown while trying to execute the Http method and stack trace: Job-7851213 Error in [Processes/References/qwerty/qwerty.xyz.process/Invoke qwerty par Status check] An IOException was thrown while trying to execute the Http method at com.myapp.plugin.share.http.client.JakartaHttpTransportDriver$RequestExecutor.run(Unknown Source) at com.myapp.pe.util.ThreadPool$ThreadPoolThread.run(Unknown Source) caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at java.net.Socket.connect(Socket.java:538) at java.net.Socket.<init>(Socket.java:434) at java.net.Socket.<init>(Socket.java:286) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:79) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:121) at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:706) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:386) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324) at com.myapp.plugin.share.http.client.JakartaHttpTransportDriver$RequestExecutor.run(Unknown Source) at com.myapp.pe.util.ThreadPool$ThreadPoolThread.run(Unknown Source)

Regards,
Abhi

2

Can Anybody please help me with this issue i want to start next log from new datestamp i tried by putting below settings in Filebeat .yml file

multiline.pattern: ^\d{4}$\ (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\ (([0-9])|([0-2][0-9])|([3][0-1]))

multiline.negate: true

multiline.match: after

Regards,
Abhi

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.