Help Needed with Logstash 1.5 OOM error

Rahul_Durrani · June 12, 2015, 6:15am

I am a newbie at logstash and I am working on setting it up for my environment, but after a few hours of running it logstash crashes with the flowing error Error:

Your application used more memory than the safety cap of 500M.
Specify -J-Xmx####m to increase it (#### = cap size in MB).
Specify -w for full OutOfMemoryError stack trace

I have used multiline, grok and mutate filters in my config file. I have also gone through all the comments on the webpage https://github.com/elastic/logstash/issues/3003 , it seems to me that multiline filter has some problem with memory leaks and hence my heap size gets exhausted. The only immediate solution I see right now is to increase my heap size and then try to tune it further, but I not really sure as to how to do that it is a parameter in logstash config or is it a setting in JVM?

Can you please let me know as to how can I Increase the heap size and Is there any other longterm solution to this problem ?

Thanks in advance ,
Rahul

colinsurprenant · June 12, 2015, 3:45pm

Hi Rahul,

Are you using the multiline filter or codec? The multiline codec has been updated to prevent OOM problem for inputs when multiline boundaries are never detected. If you are using the multiline filter I strongly suggest you switch to the multiline codec.

If this is in fact related to multiline, this mean that there is a problem detecting the multiline pattern in some way. You will have to fix this regardless.

Colin

Rahul_Durrani · June 12, 2015, 7:42pm

Hi Colin,

I am using multiline filter. How do I switch to multiline codec ?
Can I find any documentation anywhere ?

Filter that I have used

multiline {
type => "Weblogic"
pattern => "^<%{MONTH} %{MONTHDAY}, %{YEAR} %{TIME} (?:AM|PM) (CEST)>"
what => "previous"
negate =>"true"
}

Any suggestions on how to convert this ?

Thanks,
Rahul

colinsurprenant · June 12, 2015, 8:03pm

Rahul,

Can you share your current config?

Colin

magnusbaeck · June 12, 2015, 8:21pm

I am using multiline filter. How do I switch to multiline codec ?
Can I find any documentation anywhere ?

eperry · June 13, 2015, 2:49pm

I ran in to a similar issue of my filters not keeping up, my solution was to increase the number of worker thread for Logstash from the default 1 to 4, (I have lots of cores). This resolved some of my OOM so I could scale back from 8GB HEAP to 1GB. I'll check out this codec as it seems like a smarter way of working

Sometimes brute force is helpful when you have the log volume.

Rahul_Durrani · June 13, 2015, 5:00pm

Hi ,

How do I increase the number of worker threads? Where can I find this configuration ?

Thanks,
Rahul

eperry · June 13, 2015, 6:18pm

It is the flag -w # on the command line

Careful I just learned that the Multiline filter is not thread safe but the advise is to use Multiline Codec

Rahul_Durrani · June 15, 2015, 7:34am

Hi Colin,

My config file is as follows.

input 
{
  file 
     {
       type => "QAT-SOA01-OutFile"
       add_field => { server => "QAT-SOA01" }

       path=> "/u01/SOA_QAT/logs/domain_SOA_QAT01/server_soa_QAT01/server_soa_QAT01.out"
       # codec => plain { charset => "ISO-8859-1" }
     }
 file 
     {
       type => "QAT-SOA02-OutFile"
	add_field => { server => "QAT-SOA02" }
       path=> "/u01/SOA_QAT/logs/domain_SOA_QAT01/server_soa_QAT02/server_soa_QAT02.out"
       # codec => plain { charset => "ISO-8859-1" }
     }
 file 
     {
       type => "QAT-SOA03-OutFile"
	add_field => { server => "QAT-SOA03" }
       path=> "/u01/SOA_QAT/logs/domain_SOA_QAT01/server_soa_QAT03/server_soa_QAT03.out"
       # codec => plain { charset => "ISO-8859-1" }
     }
 file 
     {
       type => "QAT-SOA04-OutFile"
	add_field => { server => "QAT-SOA04" }
       path=> "/u01/SOA_QAT/logs/domain_SOA_QAT01/server_soa_QAT04/server_soa_QAT04.out"
       # codec => plain { charset => "ISO-8859-1" }
     }
}

filter  
    {
     multiline 
             {
              #type => "QAT-SOA01-OutFile"
              pattern => "^<%{MONTH} %{MONTHDAY}, %{YEAR} %{TIME} (?:AM|PM) (CEST)>"
              #  pattern => "^####"
              what => "previous"
              negate =>"true"
             }
grok
    {
     pattern => "(?m)<%{DATA:wls_timestamp}> <%{DATA:severity}> <%{DATA:wls_engine}> <%{DATA:wls_errorcode}> <%{GREEDYDATA:wls_logmessage}>"
    }
mutate
    {
    remove_tag => ["multiline"]
    }
if "Error" in [severity]
    {
   if "JMSException" in [wls_logmessage]
     {    
    mutate
          { 
          add_tag => ["Server Error" , "JMS Exception"] 
          }
     }
     }
if "Error" in [severity]
    {
   if "NullPointerException" in [wls_logmessage]
     {    
    mutate
          { 
          add_tag => ["Technical Error" , "NPE Exception"] 
	   add_field => { NOTE => "DataBase Transaction Rollback" }
          }
     }
     }
if "Error" in [severity]
    {
   if "Oracle.soa.bpel.engine.dispatch" in [wls_engine]
     {
	if "ORABPEL-05002" in [wls_logmessage]    
	  {
    mutate
          { 
          add_tag => ["Technical Error" , "Error while invocation + unhandeled fault"] 
          }
	  }
     }
     }

if "Error" in [severity]
    {
   if "oracle.soa.bpel.engine.xml" in [wls_engine]
     {    
	if "BPELFault" in [wls_logmessage]
 	{
    mutate
          { 
          add_tag => ["Technical Error" , "Invalid Value in xpath"] 
          }
     }
	}
     }

    }



output 
     {
      elasticsearch 
                   {
           # type => "QAT-SOA01-OutFile"
            host => "10.24.240.16"
            protocol=>"http"
            port => "9200"    
                   }
     }

Do you have any suggestions as to how to optimize it ?

thanks,
Rahul

Rahul_Durrani · June 15, 2015, 8:07am

Hi magnus,

On using the multiline codec my logs are not getting parsed correctly.

> My config is 
input 
{
  file {
  #type => "Weblogic"
  path=> "D:\Rahul\Greylog\server_soa_PRD01.out"
   codec => plain { charset => "ISO-8859-1" }
   codec => multiline {
      pattern => "^<%{MONTH} %{MONTHDAY}, %{YEAR} %{TIME} (?:AM|PM) (CEST)>"
        what => "previous"
		negate =>"true"
    }

  }
}
 filter {
 grok
 {
 pattern => "(?m)<%{DATA:wls_timestamp}> <%{DATA:severity}> <%{DATA:wls_engine}> <%{DATA:wls_errorcode}> <%{GREEDYDATA:wls_logmessage}>"
 }
 }

But the output I get is fragmented with tag "_grokparsefailure". My log file entries are like below:

<Apr 25, 2015 3:11:24 AM CEST> <Error> <oracle.soa.bpel.engine.xml> <BEA-000000> <
javax.xml.xpath.XPathExpressionException: javax.xml.transform.TransformerException: oramds:/deployed-composites/Rso/CreateItemJDEE1S0053FuelStockReportProvABCSImpl_rev1.0/xsl/CreateItemEBM_To_FaultMessage.xsl<Line 33, Column 258>: XML-22044: (Error) Extension function error: Error invoking 'queryDatabase':'oracle.xml.sql.OracleXMLSQLException: ORA-00936: missing expression
'
	at oracle.xml.xpath.JXPathExpression.evaluate(JXPathExpression.java:242)
	at com.collaxa.cube.xml.xpath.BPELXPathUtil.evaluate(BPELXPathUtil.java:246)
	at com.collaxa.cube.engine.ext.bpel.common.BPELWMPHelper.evalFromValue(BPELWMPHelper.java:346)
	at com.collaxa.cube.engine.ext.bpel.v1.wmp.BPEL1AssignWMP.__executeStatements(BPEL1AssignWMP.java:138)
	at com.collaxa.cube.engine.ext.bpel.common.wmp.BaseBPELActivityWMP.perform(BaseBPELActivityWMP.java:166)
	at com.collaxa.cube.engine.CubeEngine.performActivity(CubeEngine.java:2699)
	at com.collaxa.cube.engine.CubeEngine._handleWorkItem(CubeEngine.java:1190)
	at com.collaxa.cube.engine.CubeEngine.handleWorkItem(CubeEngine.java:1093)
	at com.collaxa.cube.engine.dispatch.message.instance.PerformMessageHandler.handleLocal(PerformMessageHandler.java:76)
	at com.collaxa.cube.engine.dispatch.DispatchHelper.handleLocalMessage(DispatchHelper.java:232)
	at com.collaxa.cube.engine.dispatch.DispatchHelper.sendMemory(DispatchHelper.java:311)
	at com.collaxa.cube.engine.CubeEngine.endRequest(CubeEngine.java:4621)
	at com.collaxa.cube.engine.CubeEngine.endRequest(CubeEngine.java:4552)
	at com.collaxa.cube.engine.CubeEngine._createAndInvoke(CubeEngine.java:713)
	at com.collaxa.cube.engine.CubeEngine.createAndInvoke(CubeEngine.java:560)
	at com.collaxa.cube.engine.ejb.impl.CubeEngineBean.createAndInvoke(CubeEngineBean.java:103)
	at com.collaxa.cube.engine.ejb.impl.CubeEngineBean.syncCreateAndInvokeParticipate(CubeEngineBean.java:181)
	at sun.reflect.GeneratedMethodAccessor1731.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
	at com.oracle.pitchfork.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:103)
	at oracle.security.jps.ee.ejb.JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:113)
	at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:315)
	at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:442)
	at oracle.security.jps.ee.ejb.JpsAbsInterceptor.runJaasMode(JpsAbsInterceptor.java:100)
	at oracle.security.jps.ee.ejb.JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:154)
	at oracle.security.jps.ee.ejb.JpsInterceptor.intercept(JpsInterceptor.java:113)
	at sun.reflect.GeneratedMethodAccessor1420.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
	at com.oracle.pitchfork.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:68)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
	at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at com.oracle.pitchfork.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:34)
	at weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:54)
	at com.oracle.pitchfork.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:42)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
	at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at $Proxy293.syncCreateAndInvokeParticipate(Unknown Source)
	at com.collaxa.cube.engine.ejb.impl.bpel.BPELEngineBean_51369e_ICubeEngineLocalBeanImpl.__WL_invoke(Unknown Source)
	at weblogic.ejb.container.internal.SessionLocalMethodInvoker.invoke(SessionLocalMethodInvoker.java:39)
	at com.collaxa.cube.engine.ejb.impl.bpel.BPELEngineBean_51369e_ICubeEngineLocalBeanImpl.syncCreateAndInvokeParticipate(Unknown Source)
	at com.collaxa.cube.engine.delivery.DeliveryHandler.callCreateAndInvoke(DeliveryHandler.java:911)
	at com.collaxa.cube.engine.delivery.DeliveryHandler.initialPostAnyType(DeliveryHandler.java:436)
	at com.collaxa.cube.engine.delivery.DeliveryHandler.initialPost(DeliveryHandler.java:322)
	at com.collaxa.cube.engine.delivery.DeliveryHandler.post(DeliveryHandler.java:144)
	at com.collaxa.cube.engine.ejb.impl.CubeDeliveryBean.post(CubeDeliveryBean.java:590)
	at sun.reflect.GeneratedMethodAccessor1711.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
	at com.oracle.pitchfork.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:103)
	at oracle.security.jps.ee.ejb.JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:113)
	at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:315)
	at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:442)
	at oracle.security.jps.ee.ejb.JpsAbsInterceptor.runJaasMode(JpsAbsInterceptor.java:100)
	at oracle.security.jps.ee.ejb.JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:154)
	at oracle.security.jps.ee.ejb.JpsInterceptor.intercept(JpsInterceptor.java:113)
	at sun.reflect.GeneratedMethodAccessor1420.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
	at com.oracle.pitchfork.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:68)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
	at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at com.oracle.pitchfork.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:34)
	at weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:54)
	at com.oracle.pitchfork.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:42)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
	at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at $Proxy289.post(Unknown Source)
	at com.collaxa.cube.engine.ejb.impl.bpel.BPELDeliveryBean_5k948i_ICubeDeliveryLocalBeanImpl.__WL_invoke(Unknown Source)
	at weblogic.ejb.container.internal.SessionLocalMethodInvoker.invoke(SessionLocalMethodInvoker.java:39)
	at com.collaxa.cube.engine.ejb.impl.bpel.BPELDeliveryBean_5k948i_ICubeDeliveryLocalBeanImpl.post(Unknown Source)
	at oracle.fabric.CubeServiceEngine.post(CubeServiceEngine.java:558)
	at oracle.integration.platform.blocks.mesh.AsynchronousMessageHandler.doPost(AsynchronousMessageHandler.java:142)
	at oracle.integration.platform.blocks.mesh.MessageRouter.post(MessageRouter.java:197)
	at oracle.integration.platform.blocks.mesh.MeshImpl.post(MeshImpl.java:215)
	at sun.reflect.GeneratedMethodAccessor1706.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
	at oracle.integration.platform.metrics.PhaseEventAspect.invoke(PhaseEventAspect.java:59)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at $Proxy307.post(Unknown Source)
	at oracle.integration.platform.blocks.soap.WebServiceEntryBindingComponent.doMessageProcessing(WebServiceEntryBindingComponent.java:1550)
	at oracle.integration.platform.blocks.soap.WebServiceEntryBindingComponent.processIncomingMessage(WebServiceEntryBindingComponent.java:945)
	at oracle.integration.platform.blocks.soap.FabricProvider.processMessage(FabricProvider.java:113)
	at oracle.j2ee.ws.server.provider.ProviderProcessor.doEndpointProcessing(ProviderProcessor.java:1187)
	at oracle.j2ee.ws.server.WebServiceProcessor.invokeEndpointImplementation(WebServiceProcessor.java:1112)
	at oracle.j2ee.ws.server.provider.ProviderProcessor.doRequestProcessing(ProviderProcessor.java:581)
	at oracle.j2ee.ws.server.WebServiceProcessor.processRequest(WebServiceProcessor.java:233)
	at oracle.j2ee.ws.server.WebServiceProcessor.doService(WebServiceProcessor.java:193)
	at oracle.j2ee.ws.server.WebServiceServlet.doPost(WebServiceServlet.java:485)
	at oracle.integration.platform.blocks.soap.FabricProviderServlet.doPost(FabricProviderServlet.java:528)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
	at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
	at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)
	at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
	at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
	at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:315)
	at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:442)
	at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
	at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:171)
	at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
	at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:139)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3730)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
	at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273)
	at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179)
	at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490)
	at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
	at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
>

Rahul_Durrani · June 15, 2015, 9:31am

Not able to use the multiline codec. Please see my other posts.

magnusbaeck · June 24, 2015, 6:15am

grok
{
pattern => "(?m)<%{DATA:wls_timestamp}> <%{DATA:severity}> <%{DATA:wls_engine}> <%{DATA:wls_errorcode}> <%{GREEDYDATA:wls_logmessage}>"
}

This should be:

grok {
    match => [
      "message",
      "(?m)<%{DATA:wls_timestamp}> <%{DATA:severity}> <%{DATA:wls_engine}> <%{DATA:wls_errorcode}> <%{GREEDYDATA:wls_logmessage}>"
    ]
 }

Secondly, I strongly discourage use of multiple DATA patterns. They can easily backfire and match things incorrectly.