Help Needed with Logstash 1.5 OOM error


(Rahul Durrani) #1

I am a newbie at logstash and I am working on setting it up for my environment, but after a few hours of running it logstash crashes with the flowing error Error:

Your application used more memory than the safety cap of 500M.
Specify -J-Xmx####m to increase it (#### = cap size in MB).
Specify -w for full OutOfMemoryError stack trace

I have used multiline, grok and mutate filters in my config file. I have also gone through all the comments on the webpage https://github.com/elastic/logstash/issues/3003 , it seems to me that multiline filter has some problem with memory leaks and hence my heap size gets exhausted. The only immediate solution I see right now is to increase my heap size and then try to tune it further, but I not really sure as to how to do that it is a parameter in logstash config or is it a setting in JVM?

Can you please let me know as to how can I Increase the heap size and Is there any other longterm solution to this problem ?

Thanks in advance ,
Rahul


(Colin Surprenant) #2

Hi Rahul,

Are you using the multiline filter or codec? The multiline codec has been updated to prevent OOM problem for inputs when multiline boundaries are never detected. If you are using the multiline filter I strongly suggest you switch to the multiline codec.

If this is in fact related to multiline, this mean that there is a problem detecting the multiline pattern in some way. You will have to fix this regardless.

Colin


(Rahul Durrani) #3

Hi Colin,

I am using multiline filter. How do I switch to multiline codec ?
Can I find any documentation anywhere ?

Filter that I have used

multiline {
type => "Weblogic"
pattern => "^<%{MONTH} %{MONTHDAY}, %{YEAR} %{TIME} (?:AM|PM) (CEST)>"
what => "previous"
negate =>"true"
}

Any suggestions on how to convert this ?

Thanks,
Rahul


(Colin Surprenant) #4

Rahul,

Can you share your current config?

Colin


(Magnus Bäck) #5

I am using multiline filter. How do I switch to multiline codec ?
Can I find any documentation anywhere ?

https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html


(Ed) #6

I ran in to a similar issue of my filters not keeping up, my solution was to increase the number of worker thread for Logstash from the default 1 to 4, (I have lots of cores). This resolved some of my OOM so I could scale back from 8GB HEAP to 1GB. I'll check out this codec as it seems like a smarter way of working

Sometimes brute force is helpful when you have the log volume. :smile:


(Rahul Durrani) #7

Hi ,

How do I increase the number of worker threads? Where can I find this configuration ?

Thanks,
Rahul


(Ed) #8

It is the flag -w # on the command line

Careful I just learned that the Multiline filter is not thread safe but the advise is to use Multiline Codec


(Rahul Durrani) #9

Hi Colin,

My config file is as follows.

input 
{
  file 
     {
       type => "QAT-SOA01-OutFile"
       add_field => { server => "QAT-SOA01" }

       path=> "/u01/SOA_QAT/logs/domain_SOA_QAT01/server_soa_QAT01/server_soa_QAT01.out"
       # codec => plain { charset => "ISO-8859-1" }
     }
 file 
     {
       type => "QAT-SOA02-OutFile"
	add_field => { server => "QAT-SOA02" }
       path=> "/u01/SOA_QAT/logs/domain_SOA_QAT01/server_soa_QAT02/server_soa_QAT02.out"
       # codec => plain { charset => "ISO-8859-1" }
     }
 file 
     {
       type => "QAT-SOA03-OutFile"
	add_field => { server => "QAT-SOA03" }
       path=> "/u01/SOA_QAT/logs/domain_SOA_QAT01/server_soa_QAT03/server_soa_QAT03.out"
       # codec => plain { charset => "ISO-8859-1" }
     }
 file 
     {
       type => "QAT-SOA04-OutFile"
	add_field => { server => "QAT-SOA04" }
       path=> "/u01/SOA_QAT/logs/domain_SOA_QAT01/server_soa_QAT04/server_soa_QAT04.out"
       # codec => plain { charset => "ISO-8859-1" }
     }
}

filter  
    {
     multiline 
             {
              #type => "QAT-SOA01-OutFile"
              pattern => "^<%{MONTH} %{MONTHDAY}, %{YEAR} %{TIME} (?:AM|PM) (CEST)>"
              #  pattern => "^####"
              what => "previous"
              negate =>"true"
             }
grok
    {
     pattern => "(?m)<%{DATA:wls_timestamp}> <%{DATA:severity}> <%{DATA:wls_engine}> <%{DATA:wls_errorcode}> <%{GREEDYDATA:wls_logmessage}>"
    }
mutate
    {
    remove_tag => ["multiline"]
    }
if "Error" in [severity]
    {
   if "JMSException" in [wls_logmessage]
     {    
    mutate
          { 
          add_tag => ["Server Error" , "JMS Exception"] 
          }
     }
     }
if "Error" in [severity]
    {
   if "NullPointerException" in [wls_logmessage]
     {    
    mutate
          { 
          add_tag => ["Technical Error" , "NPE Exception"] 
	   add_field => { NOTE => "DataBase Transaction Rollback" }
          }
     }
     }
if "Error" in [severity]
    {
   if "Oracle.soa.bpel.engine.dispatch" in [wls_engine]
     {
	if "ORABPEL-05002" in [wls_logmessage]    
	  {
    mutate
          { 
          add_tag => ["Technical Error" , "Error while invocation + unhandeled fault"] 
          }
	  }
     }
     }

if "Error" in [severity]
    {
   if "oracle.soa.bpel.engine.xml" in [wls_engine]
     {    
	if "BPELFault" in [wls_logmessage]
 	{
    mutate
          { 
          add_tag => ["Technical Error" , "Invalid Value in xpath"] 
          }
     }
	}
     }

    }



output 
     {
      elasticsearch 
                   {
           # type => "QAT-SOA01-OutFile"
            host => "10.24.240.16"
            protocol=>"http"
            port => "9200"    
                   }
     }

Do you have any suggestions as to how to optimize it ?

thanks,
Rahul


(Rahul Durrani) #10

Hi magnus,

On using the multiline codec my logs are not getting parsed correctly.

> My config is 
input 
{
  file {
  #type => "Weblogic"
  path=> "D:\Rahul\Greylog\server_soa_PRD01.out"
   codec => plain { charset => "ISO-8859-1" }
   codec => multiline {
      pattern => "^<%{MONTH} %{MONTHDAY}, %{YEAR} %{TIME} (?:AM|PM) (CEST)>"
        what => "previous"
		negate =>"true"
    }

  }
}
 filter {
 grok
 {
 pattern => "(?m)<%{DATA:wls_timestamp}> <%{DATA:severity}> <%{DATA:wls_engine}> <%{DATA:wls_errorcode}> <%{GREEDYDATA:wls_logmessage}>"
 }
 }

But the output I get is fragmented with tag "_grokparsefailure". My log file entries are like below:

<Apr 25, 2015 3:11:24 AM CEST> <Error> <oracle.soa.bpel.engine.xml> <BEA-000000> <
javax.xml.xpath.XPathExpressionException: javax.xml.transform.TransformerException: oramds:/deployed-composites/Rso/CreateItemJDEE1S0053FuelStockReportProvABCSImpl_rev1.0/xsl/CreateItemEBM_To_FaultMessage.xsl<Line 33, Column 258>: XML-22044: (Error) Extension function error: Error invoking 'queryDatabase':'oracle.xml.sql.OracleXMLSQLException: ORA-00936: missing expression
'
	at oracle.xml.xpath.JXPathExpression.evaluate(JXPathExpression.java:242)
	at com.collaxa.cube.xml.xpath.BPELXPathUtil.evaluate(BPELXPathUtil.java:246)
	at com.collaxa.cube.engine.ext.bpel.common.BPELWMPHelper.evalFromValue(BPELWMPHelper.java:346)
	at com.collaxa.cube.engine.ext.bpel.v1.wmp.BPEL1AssignWMP.__executeStatements(BPEL1AssignWMP.java:138)
	at com.collaxa.cube.engine.ext.bpel.common.wmp.BaseBPELActivityWMP.perform(BaseBPELActivityWMP.java:166)
	at com.collaxa.cube.engine.CubeEngine.performActivity(CubeEngine.java:2699)
	at com.collaxa.cube.engine.CubeEngine._handleWorkItem(CubeEngine.java:1190)
	at com.collaxa.cube.engine.CubeEngine.handleWorkItem(CubeEngine.java:1093)
	at com.collaxa.cube.engine.dispatch.message.instance.PerformMessageHandler.handleLocal(PerformMessageHandler.java:76)
	at com.collaxa.cube.engine.dispatch.DispatchHelper.handleLocalMessage(DispatchHelper.java:232)
	at com.collaxa.cube.engine.dispatch.DispatchHelper.sendMemory(DispatchHelper.java:311)
	at com.collaxa.cube.engine.CubeEngine.endRequest(CubeEngine.java:4621)
	at com.collaxa.cube.engine.CubeEngine.endRequest(CubeEngine.java:4552)
	at com.collaxa.cube.engine.CubeEngine._createAndInvoke(CubeEngine.java:713)
	at com.collaxa.cube.engine.CubeEngine.createAndInvoke(CubeEngine.java:560)
	at com.collaxa.cube.engine.ejb.impl.CubeEngineBean.createAndInvoke(CubeEngineBean.java:103)
	at com.collaxa.cube.engine.ejb.impl.CubeEngineBean.syncCreateAndInvokeParticipate(CubeEngineBean.java:181)
	at sun.reflect.GeneratedMethodAccessor1731.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
	at com.oracle.pitchfork.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:103)
	at oracle.security.jps.ee.ejb.JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:113)
	at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:315)
	at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:442)
	at oracle.security.jps.ee.ejb.JpsAbsInterceptor.runJaasMode(JpsAbsInterceptor.java:100)
	at oracle.security.jps.ee.ejb.JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:154)
	at oracle.security.jps.ee.ejb.JpsInterceptor.intercept(JpsInterceptor.java:113)
	at sun.reflect.GeneratedMethodAccessor1420.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
	at com.oracle.pitchfork.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:68)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
	at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at com.oracle.pitchfork.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:34)
	at weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:54)
	at com.oracle.pitchfork.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:42)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
	at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at $Proxy293.syncCreateAndInvokeParticipate(Unknown Source)
	at com.collaxa.cube.engine.ejb.impl.bpel.BPELEngineBean_51369e_ICubeEngineLocalBeanImpl.__WL_invoke(Unknown Source)
	at weblogic.ejb.container.internal.SessionLocalMethodInvoker.invoke(SessionLocalMethodInvoker.java:39)
	at com.collaxa.cube.engine.ejb.impl.bpel.BPELEngineBean_51369e_ICubeEngineLocalBeanImpl.syncCreateAndInvokeParticipate(Unknown Source)
	at com.collaxa.cube.engine.delivery.DeliveryHandler.callCreateAndInvoke(DeliveryHandler.java:911)
	at com.collaxa.cube.engine.delivery.DeliveryHandler.initialPostAnyType(DeliveryHandler.java:436)
	at com.collaxa.cube.engine.delivery.DeliveryHandler.initialPost(DeliveryHandler.java:322)
	at com.collaxa.cube.engine.delivery.DeliveryHandler.post(DeliveryHandler.java:144)
	at com.collaxa.cube.engine.ejb.impl.CubeDeliveryBean.post(CubeDeliveryBean.java:590)
	at sun.reflect.GeneratedMethodAccessor1711.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
	at com.oracle.pitchfork.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:103)
	at oracle.security.jps.ee.ejb.JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:113)
	at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:315)
	at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:442)
	at oracle.security.jps.ee.ejb.JpsAbsInterceptor.runJaasMode(JpsAbsInterceptor.java:100)
	at oracle.security.jps.ee.ejb.JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:154)
	at oracle.security.jps.ee.ejb.JpsInterceptor.intercept(JpsInterceptor.java:113)
	at sun.reflect.GeneratedMethodAccessor1420.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
	at com.oracle.pitchfork.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:68)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
	at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at com.oracle.pitchfork.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:34)
	at weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:54)
	at com.oracle.pitchfork.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:42)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
	at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
	at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at $Proxy289.post(Unknown Source)
	at com.collaxa.cube.engine.ejb.impl.bpel.BPELDeliveryBean_5k948i_ICubeDeliveryLocalBeanImpl.__WL_invoke(Unknown Source)
	at weblogic.ejb.container.internal.SessionLocalMethodInvoker.invoke(SessionLocalMethodInvoker.java:39)
	at com.collaxa.cube.engine.ejb.impl.bpel.BPELDeliveryBean_5k948i_ICubeDeliveryLocalBeanImpl.post(Unknown Source)
	at oracle.fabric.CubeServiceEngine.post(CubeServiceEngine.java:558)
	at oracle.integration.platform.blocks.mesh.AsynchronousMessageHandler.doPost(AsynchronousMessageHandler.java:142)
	at oracle.integration.platform.blocks.mesh.MessageRouter.post(MessageRouter.java:197)
	at oracle.integration.platform.blocks.mesh.MeshImpl.post(MeshImpl.java:215)
	at sun.reflect.GeneratedMethodAccessor1706.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
	at oracle.integration.platform.metrics.PhaseEventAspect.invoke(PhaseEventAspect.java:59)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at $Proxy307.post(Unknown Source)
	at oracle.integration.platform.blocks.soap.WebServiceEntryBindingComponent.doMessageProcessing(WebServiceEntryBindingComponent.java:1550)
	at oracle.integration.platform.blocks.soap.WebServiceEntryBindingComponent.processIncomingMessage(WebServiceEntryBindingComponent.java:945)
	at oracle.integration.platform.blocks.soap.FabricProvider.processMessage(FabricProvider.java:113)
	at oracle.j2ee.ws.server.provider.ProviderProcessor.doEndpointProcessing(ProviderProcessor.java:1187)
	at oracle.j2ee.ws.server.WebServiceProcessor.invokeEndpointImplementation(WebServiceProcessor.java:1112)
	at oracle.j2ee.ws.server.provider.ProviderProcessor.doRequestProcessing(ProviderProcessor.java:581)
	at oracle.j2ee.ws.server.WebServiceProcessor.processRequest(WebServiceProcessor.java:233)
	at oracle.j2ee.ws.server.WebServiceProcessor.doService(WebServiceProcessor.java:193)
	at oracle.j2ee.ws.server.WebServiceServlet.doPost(WebServiceServlet.java:485)
	at oracle.integration.platform.blocks.soap.FabricProviderServlet.doPost(FabricProviderServlet.java:528)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
	at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
	at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)
	at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
	at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
	at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:315)
	at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:442)
	at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
	at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:171)
	at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
	at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:139)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3730)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
	at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273)
	at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179)
	at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490)
	at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
	at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
>

(Rahul Durrani) #11

Not able to use the multiline codec. Please see my other posts.


(Magnus Bäck) #12

grok
{
pattern => "(?m)<%{DATA:wls_timestamp}> <%{DATA:severity}> <%{DATA:wls_engine}> <%{DATA:wls_errorcode}> <%{GREEDYDATA:wls_logmessage}>"
}

This should be:

grok {
    match => [
      "message",
      "(?m)<%{DATA:wls_timestamp}> <%{DATA:severity}> <%{DATA:wls_engine}> <%{DATA:wls_errorcode}> <%{GREEDYDATA:wls_logmessage}>"
    ]
 }

Secondly, I strongly discourage use of multiple DATA patterns. They can easily backfire and match things incorrectly.


(system) #13