Logs parsing for multiline logs

Elastic Stack Logstash
1

Hello Team,

Below is the sample logs

| 2025-06-24 07:47:24 |ERROR| request-worker-13 | com.orsyp.central.ldap.MD5Login | Authentication Exception
javax.naming.AuthenticationException: [LDAP: error code 49 - 8009030C: LdapErr: xxxx, comment: AcceptSecurityContext error, data 775, abcd ]
      at com.sun.jndi.ldap.LdapCtx.mapErrorCode(Unknown Source)
      at com.sun.jndi.ldap.LdapCtx.processReturnCode(Unknown Source)
      at com.sun.jndi.ldap.LdapCtx.processReturnCode(Unknown Source)
      at com.sun.jndi.ldap.LdapCtx.connect(Unknown Source)
      at com.sun.jndi.ldap.LdapCtx.<init>(Unknown Source)
      at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(Unknown Source)
      at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(Unknown Source)
      at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(Unknown Source)
      at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(Unknown Source)
      at javax.naming.spi.NamingManager.getInitialContext(Unknown Source)
      at javax.naming.InitialContext.getDefaultInitCtx(Unknown Source)
      at javax.naming.InitialContext.init(Unknown Source)
      at javax.naming.InitialContext.<init>(Unknown Source)
      at javax.naming.directory.InitialDirContext.<init>(Unknown Source)
      at com.orsyp.central.ldap.MD5Login.login(MD5Login.java:60)
      at com.orsyp.central.ldap.LDAPManagerImpl.authenticate(LDAPManagerImpl.java:380)
      at com.orsyp.central.server.AuthentificationStdImpl.doLDAPAuthentication(AuthentificationStdImpl.java:138)
      at com.orsyp.central.server.AuthentificationStdImpl.authSocket(AuthentificationStdImpl.java:217)
      at com.orsyp.comm.server.NIOBasedSocket.authentification(NIOBasedSocket.java:216)
      at com.orsyp.comm.server.NIOBasedSocket.readStream(NIOBasedSocket.java:178)
      at com.orsyp.central.server.UniWorker.doWork(UniWorker.java:194)
      at com.orsyp.central.server.CentralServerAdapter$1.run(CentralServerAdapter.java:273)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)

and i am using following grok pattern `| %{TIMESTAMP_ISO8601:timestamp} |%{LOGLEVEL:level}| %{DATA:request} | %{DATA:package_name} | %{GREEDYDATA:data}`

Below are the fields extracted using grok

{
    "timestamp": "2025-06-24 07:47:24",
    "level": "ERROR",
    "request": "request-worker-13",
    "package_name": "com.orsyp.central.ldap.MD5Login",
    "data": "Authentication Exception"
  }

In data field i want to keep Authentication Exception
javax.naming.AuthenticationException: [LDAP: error code 49 - 8009030C: LdapErr: xxxx, comment: AcceptSecurityContext error, data 775, abcd ] in addition to that i want capture error_desciption and the value will be

Authentication Exception
javax.naming.AuthenticationException: [LDAP: error code 49 - 8009030C: LdapErr: xxxx, comment: AcceptSecurityContext error, data 775, abcd ]
      at com.sun.jndi.ldap.LdapCtx.mapErrorCode(Unknown Source)
      at com.sun.jndi.ldap.LdapCtx.processReturnCode(Unknown Source)
      at com.sun.jndi.ldap.LdapCtx.processReturnCode(Unknown Source)
      at com.sun.jndi.ldap.LdapCtx.connect(Unknown Source)
      at com.sun.jndi.ldap.LdapCtx.<init>(Unknown Source)
      at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(Unknown Source)
      at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(Unknown Source)
      at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(Unknown Source)
      at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(Unknown Source)
      at javax.naming.spi.NamingManager.getInitialContext(Unknown Source)
      at javax.naming.InitialContext.getDefaultInitCtx(Unknown Source)
      at javax.naming.InitialContext.init(Unknown Source)
      at javax.naming.InitialContext.<init>(Unknown Source)
      at javax.naming.directory.InitialDirContext.<init>(Unknown Source)
      at com.orsyp.central.ldap.MD5Login.login(MD5Login.java:60)
      at com.orsyp.central.ldap.LDAPManagerImpl.authenticate(LDAPManagerImpl.java:380)
      at com.orsyp.central.server.AuthentificationStdImpl.doLDAPAuthentication(AuthentificationStdImpl.java:138)
      at com.orsyp.central.server.AuthentificationStdImpl.authSocket(AuthentificationStdImpl.java:217)
      at com.orsyp.comm.server.NIOBasedSocket.authentification(NIOBasedSocket.java:216)
      at com.orsyp.comm.server.NIOBasedSocket.readStream(NIOBasedSocket.java:178)
      at com.orsyp.central.server.UniWorker.doWork(UniWorker.java:194)
      at com.orsyp.central.server.CentralServerAdapter$1.run(CentralServerAdapter.java:273)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)

How can i achieve this .

2

Step one is to capture the entire exception and stack trace as a single event using a multiline codec on the input (if the file is being read by filebeat then do this in filebeat, not in logstash). You could try something like

multiline {
    pattern => "^\| \d{4}-\d{2}-" 
    negate => true 
    what => previous 
    auto_flush_interval => 2
}

But depending on the structure of other error messages in the log you may need to tune that. Since you want [error_description] to include the entire error message you should do that in your first grok.

grok { match => { "message" => "^\| %{TIMESTAMP_ISO8601:timestamp} \|%{LOGLEVEL:level}\| %{DATA:request} \| %{DATA:package_name} \| %{GREEDYDATA:error_description}" } remove_field => [ "message" ] }

Then parse [error_description] to capture the first two lines in another field

 grok { match => { "error_description" => "^(?<data>[^\n]*\n[^\n]*\n)" } }

That will get you

            "level" => "ERROR",
        "timestamp" => "2025-06-24 07:47:24",
          "request" => "request-worker-13",
     "package_name" => "com.orsyp.central.ldap.MD5Login",
"error_description" => "Authentication Exception\njavax.naming.AuthenticationException: [LDAP: error code 49 - 8009030C: LdapErr: xxxx, comment: AcceptSecurityContext error, data 775, abcd ]\n      at com.sun.jndi.ldap.LdapCtx.mapErrorCode(Unknown Source)\n      at com.sun.jndi.ldap.LdapCtx.processReturnCode(Unknown Source)\n      at com.sun.jndi.ldap.LdapCtx.processReturnCode(Unknown Source)\n      at com.sun.jndi.ldap.LdapCtx.connect(Unknown Source)\n      at com.sun.jndi.ldap.LdapCtx.<init>(Unknown Source)\n      at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(Unknown Source)\n      at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(Unknown Source)\n      at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(Unknown Source)\n      at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(Unknown Source)\n      at javax.naming.spi.NamingManager.getInitialContext(Unknown Source)\n      at javax.naming.InitialContext.getDefaultInitCtx(Unknown Source)\n      at javax.naming.InitialContext.init(Unknown Source)\n      at javax.naming.InitialContext.<init>(Unknown Source)\n      at javax.naming.directory.InitialDirContext.<init>(Unknown Source)\n      at com.orsyp.central.ldap.MD5Login.login(MD5Login.java:60)\n      at com.orsyp.central.ldap.LDAPManagerImpl.authenticate(LDAPManagerImpl.java:380)\n      at com.orsyp.central.server.AuthentificationStdImpl.doLDAPAuthentication(AuthentificationStdImpl.java:138)\n      at com.orsyp.central.server.AuthentificationStdImpl.authSocket(AuthentificationStdImpl.java:217)\n      at com.orsyp.comm.server.NIOBasedSocket.authentification(NIOBasedSocket.java:216)\n      at com.orsyp.comm.server.NIOBasedSocket.readStream(NIOBasedSocket.java:178)\n      at com.orsyp.central.server.UniWorker.doWork(UniWorker.java:194)\n      at com.orsyp.central.server.CentralServerAdapter$1.run(CentralServerAdapter.java:273)\n      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n      at java.lang.Thread.run(Unknown Source)",
             "data" => "Authentication Exception\njavax.naming.AuthenticationException: [LDAP: error code 49 - 8009030C: LdapErr: xxxx, comment: AcceptSecurityContext error, data 775, abcd ]\n"
1 Like