Log parsing issue

Divya_Bansal · November 16, 2021, 10:53am

I am having a log in the following format:-

{"@timestamp":"2021-08-04T09:57:25.141Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.6.3"},"log":{"offset":10413931,"file":{"path":"api/api.log"}},"message":"[ERROR] 2021-10-18T22:36:04.672 [http-nio2-8080-exec-48] [FTDS] deployment-75bf886778-gj8hv - [i.i.i.a.e.f.s.ManageForm] :: RuntimeException is caught with error code: ITB-EXEC2003 and DevMessage: null and UserMessage: Due to some technical error not able to process.Please check corresponding class 
	   and error code ITB-EXEC2003 and stacktrace: java.lang.NullPointerException
	at in.it.ic.api.filing.form.FormData.validateAndSet(FormData.java:858)
	at in.it.ic.api.filing.form.FormData.setFeilds(FormData.java:822)
	at jdk.internal.reflect.GeneratedMethodAccessor304.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.transaction.interceptor.TransactionInterceptor$$Lambda$510/00000000046D7C60.proceedWithInvocation(Unknown Source)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:366)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:99)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
	at com.sun.proxy.$Proxy135.serve(Unknown Source)
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)
	at jdk.internal.reflect.GeneratedMethodAccessor247.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689)
	at jdk.internal.reflect.GeneratedMethodAccessor246.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:888)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:652)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Thread.java:836)
 and the possible root cause is java.lang.NullPointerException at 1634576764672 with objectarray values asnull[#]","input":{"type":"log"},"ecs":{"version":"1.4.0"},"host":{"mac":["/"],"hostname":"boot","architecture":"ppc64le","os":{"codename":"Maipo","platform":"rhel","version":"7.6 (Maipo)","family":"redhat","name":"Red Hat Enterprise Linux Server","kernel":"4.14.0-115.13.1.el7a.ppc64le"},"id":"4994b06b9b4248dd81e0c113f2221e54","name":"dcplicpboot","containerized":false,"ip":["/"]},"agent":{"version":"7.6.3","type":"filebeat","ephemeral_id":"e249728c-21f3-4ea6-be0f-b5ce5ff7447f","hostname":"boot","id":"6bd6e68b-3f3e-4985-8738-08b3488fffd7"}}

I wanted to parse the multiline message into single line in logstash. can anyone help me to do this?

Wolfram_Haussig · November 16, 2021, 11:28am

Hi,

What do you mean by parse the multiline message into single line? You want to remove the line breaks?

Best regards
Wolfram

Divya_Bansal · November 16, 2021, 12:00pm

yes, all the different lines of stack trace should be considered as single line.

Wolfram_Haussig · November 16, 2021, 12:10pm

Although I have not tried the solution, I found one on stackoverflow: elasticsearch - How to remove Newline from a log in filter block using mutate gsub - Stack Overflow

mutate{
    gsub => [ "message", "\\n", "" ]
  }

Divya_Bansal · November 16, 2021, 1:19pm

not worked

Badger · November 16, 2021, 6:00pm

You would use a multiline codec to combine the lines of the stack trace with the error message. The documentation has an example that shows how to do that for Java stack traces.

system · December 14, 2021, 6:01pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Stack trace parsing issue Logstash	10	1202	November 22, 2021
Aggregate error Logstash	2	636	September 8, 2021
Parse a Null Pointer Exception Logs in Logstash Logstash	1	451	June 15, 2020
Logstash not parsing metadata from logfile Logstash	2	324	February 5, 2021
Multiline log parsing - Filebeat, Logstash Logstash	2	377	July 15, 2020

Log parsing issue

Related topics