[Logstash 6.5.2] Problem parsing long messages

[Fabio-sama (Fabio)]

Hi there,

I'm using Logstash to parse a java json log which may have a stacktrace nested field. I wanted to use json filter to automatically parse the input (since it is a json with possible nested jsons).
Now, if I apply a simple pipeline like

input {
    stdin{}
}

filter {
    json {
        source => "message"
    }
}

output {
    stdout {
        codec => "rubydebug"
    }
}

it beautifully parses simple inputs like
{ "field1":"test1", "field2":"test2", "nested_field": { "nested1":["1",2,{ "testobj": "newtest" }], "nested2":"tnest2" } }, recognizing nested fields and arrays:

Though, when I insert a more complicated json (like the nested stacktrace present in the real log)

{ "exception":{"stacktrace": "java.lang.IllegalStateException: Impossibile Stabilire nodo primario: ORA-04063: package body LCP1.AGENT_REGISTRYNODES has errors\nORA-06508: PL/SQL: could not find program unit being called: LCP1.AGENT_REGISTRYNODES\nORA-06512: at line 1\n\n\tat it.ltm.mlt.common.scheduler.strategy.MasterNodeStrategy.canExecute(MasterNodeStrategy.java:46)\n\tat it.ltm.mlt.common.scheduler.AbstractScheduler.isPrimary(AbstractScheduler.java:70)\n\tat it.ltm.mlt.common.scheduler.AbstractScheduler.ejbTimeout(AbstractScheduler.java:57)\n\tat sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)\n\tat java.lang.reflect.Method.invoke(Method.java:597)\n\tat com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)\n\tat com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)\n\tat com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)\n\tat com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)\n\tat com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)\n\tat com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)\n\tat com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)\n\tat com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)\n\tat com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)\n\tat com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:37)\n\tat weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:54)\n\tat com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:50)\n\tat com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)\n\tat com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)\n\tat com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)\n\tat com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)\n\tat com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)\n\tat com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)\n\tat com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)\n\tat $Proxy79.ejbTimeout(Unknown Source)\n\tat weblogic.ejb.container.timer.TimerImpl.timerExpired(TimerImpl.java:301)\n\tat weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)\n\tat weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)\n\tat weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)\n\tat weblogic.work.ExecuteThread.run(ExecuteThread.java:176)\n\tCaused by: java.sql.SQLException: ORA-04063: package body LCP1.AGENT_REGISTRYNODES has errors\nORA-06508: PL/SQL: could not find program unit being called: LCP1.AGENT_REGISTRYNODES\nORA-06512: at line 1\n\n\tat oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440)\n\tat oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)\n\tat oracle.jdbc.driver.T4C8Oall_added_to_make_it_longer"} }

It immediately returns a _jsonparsefailure.

This happens because it basically truncates the message (if you see the original json the last part is _added_to_make_it_longer but in the image you can only see _add) and not finding the closing curly brackets it treats it like an invalid json.

In fact, if I remove the _added_to_make_it_longer final part, it works flawlessly

Same thing happens with the grok filter, it returns _grokparsefailure if I try to grok a line which exceeds 4095 characters. I thought this limit was increased in LS versions >5.1. I'm using the 6.5.2 and I still see this error.

Why? And more importantly, which is the workaround?

Thanks

[Fabio-sama (Fabio)]

I have seen this one. Input STDIN 4096 character input limit?

Isn't there any way to increase that limit? Also, is it possible it happens with other types of input too (like redis?)

Thanks