Hi,
I have an application log, I am reading logs from filebeat and forwarding it to logstash.
But I have log in 2 different type of multiline patterns. One is for request begin - to - response end, and in between it has java exception.
Here is my log file.
2020-04-19 21:49:58 DEBUG LoggingRequestInterceptor:28 - ============================request begin===========================================
2020-04-19 21:49:58 DEBUG LoggingRequestInterceptor:29 - URI : https://bone01.bandhan.local/Bone/MobileDevice/1.0/1234/
2020-04-19 21:49:58 DEBUG LoggingRequestInterceptor:30 - Method : POST
2020-04-19 21:49:58 DEBUG LoggingRequestInterceptor:31 - Headers : [Accept:"application/json, application/*+json", Content-Type:"application/json", Content-Length:"265"]
2020-04-19 21:49:58 DEBUG LoggingRequestInterceptor:32 - Request body: {"ApplicationId":"8","TriggeredByUserId":"","ActionId":472,"SqlScriptList":[{"ScriptId":0,"AddedOn":"2020-04-19T21:49:58.485+05:30","PkeyId":null,"ArgumentListLite":[{"Fn":"@deviceId","Fv":"123456781"},{"Fn":"@appId","Fv":"123456781"}]}],"ResponseStatus":"FAILURE"}
2020-04-19 21:49:58 DEBUG LoggingRequestInterceptor:33 - ==========================request end===============================================
2020-04-19 21:50:00 DEBUG LoggingRequestInterceptor:43 - ============================response begin==========================================
2020-04-19 21:50:00 DEBUG LoggingRequestInterceptor:44 - Status code : 200 OK
2020-04-19 21:50:00 DEBUG LoggingRequestInterceptor:45 - Status text : OK
2020-04-19 21:50:00 DEBUG LoggingRequestInterceptor:46 - Headers : [Cache-Control:"no-cache", Pragma:"no-cache", Content-Type:"application/json; charset=utf-8", Expires:"-1", Server:"Microsoft-IIS/10.0", X-AspNet-Version:"4.0.30319", X-Powered-By:"ASP.NET", Date:"Sun, 19 Apr 2020 17:13:25 GMT", Content-Length:"522"]
2020-04-19 21:50:00 DEBUG LoggingRequestInterceptor:47 - Response body: "{\"ApplicationId\":0,\"TriggeredByUserId\":\"\",\"ActionId\":0,\"ActionMethodName\":\"\",\"ActionNameSpace\":\"\",\"GroupMethodName\":\"\",\"GroupNameSpace\":\"\",\"NoOfArguments\":0,\"RequestType\":\"\",\"MethodArg\":[],\"DbTuple\":[],\"SqlScriptList\":[],\"Base64ObjectString\":\"\",\"DeviceTypeId\":0,\"DeviceId\":\"\",\"DbTupleLite\":[],\"ResponseStatus\":\"FAILURE\",\"ErrorMessage\":\"Invalid object name 'DB_MICROBANKING.DBO.T_MBN_DSC_SUPER_USER'.\",\"ErrorDetail\":\"\",\"ErrorCode\":\"\",\"ErrorLocation\":\"\"}"
2020-04-19 21:50:00 DEBUG LoggingRequestInterceptor:48 - =======================response end=================================================
2020-04-23 09:36:39 ERROR CustomResponseEntityExceptionHandler:77 - ERROR PROCESSING REQUEST-/microbanking/transactionsMS/withdraw-Invalid biometric data.
com.iexceed.bandhan.microfin.core.exception.ValidationException: Invalid biometric data.
at com.iexceed.bandhan.microfin.transactions.controller.TransactionsController.withdrawAPI(TransactionsController.java:96)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282)
at org.springframework.cloud.context.scope.GenericScope$LockedScopedProxyFactoryBean.invoke(GenericScope.java:499)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689)
at com.iexceed.bandhan.microfin.transactions.controller.TransactionsController$$EnhancerBySpringCGLIB$$e6b19c88.withdrawAPI(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:888)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:108)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1598)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
2020-04-19 21:00:58 DEBUG LoggingRequestInterceptor:28 - ============================request begin===========================================
2020-04-19 21:00:58 DEBUG LoggingRequestInterceptor:29 - URI : https://bone01.bandhan.local/Bone/MobileDevice/1.0/1234/
2020-04-19 21:49:58 DEBUG LoggingRequestInterceptor:30 - Method : POST
2020-04-19 21:49:58 DEBUG LoggingRequestInterceptor:31 - Headers : [Accept:"application/json, application/*+json", Content-Type:"application/json", Content-Length:"265"]
2020-04-19 21:49:58 DEBUG LoggingRequestInterceptor:32 - Request body: {"ApplicationId":"8","TriggeredByUserId":"","ActionId":472,"SqlScriptList":[{"ScriptId":0,"AddedOn":"2020-04-19T21:49:58.485+05:30","PkeyId":null,"ArgumentListLite":[{"Fn":"@deviceId","Fv":"123456781"},{"Fn":"@appId","Fv":"123456781"}]}],"ResponseStatus":"FAILURE"}
2020-04-19 21:49:58 DEBUG LoggingRequestInterceptor:33 - ==========================request end===============================================
2020-04-19 21:50:00 DEBUG LoggingRequestInterceptor:43 - ============================response begin==========================================
2020-04-19 21:50:00 DEBUG LoggingRequestInterceptor:44 - Status code : 200 OK
2020-04-19 21:50:00 DEBUG LoggingRequestInterceptor:45 - Status text : OK
2020-04-19 21:50:00 DEBUG LoggingRequestInterceptor:46 - Headers : [Cache-Control:"no-cache", Pragma:"no-cache", Content-Type:"application/json; charset=utf-8", Expires:"-1", Server:"Microsoft-IIS/10.0", X-AspNet-Version:"4.0.30319", X-Powered-By:"ASP.NET", Date:"Sun, 19 Apr 2020 17:13:25 GMT", Content-Length:"522"]
2020-04-19 21:50:00 DEBUG LoggingRequestInterceptor:47 - Response body: "{\"ApplicationId\":0,\"TriggeredByUserId\":\"\",\"ActionId\":0,\"ActionMethodName\":\"\",\"ActionNameSpace\":\"\",\"GroupMethodName\":\"\",\"GroupNameSpace\":\"\",\"NoOfArguments\":0,\"RequestType\":\"\",\"MethodArg\":[],\"DbTuple\":[],\"SqlScriptList\":[],\"Base64ObjectString\":\"\",\"DeviceTypeId\":0,\"DeviceId\":\"\",\"DbTupleLite\":[],\"ResponseStatus\":\"FAILURE\",\"ErrorMessage\":\"Invalid object name 'DB_MICROBANKING.DBO.T_MBN_DSC_SUPER_USER'.\",\"ErrorDetail\":\"\",\"ErrorCode\":\"\",\"ErrorLocation\":\"\"}"
2020-04-19 21:50:00 DEBUG LoggingRequestInterceptor:48 - =======================response end=================================================
and below are my filebeat patterns for
-
Request begin ------- to -------response end it will consider one event and it is working
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2} DEBUG LoggingRequestInterceptor:[0-9]{2} - ============================request begin' multiline.negate: true multiline.match: after -
Pattern for java exception.
multiline.pattern: '^[[:space:]]' multiline.negate: false multiline.match: after
Note:
When running individual pattern it works fine, but my query is I want to combine two patterns, but I am not able to do it.
Please help me with this.
Thanks,
Tahseen.