How to parse the wrapper logs which contains timestamp value at each line

[sai7276p (Sai)]

I have created grok pattern for single line entries, but I just want to remove the timestamp and extra fields before the java stack trace lines,,, to register a full stack trace to a single field 'MSG'

"(%{LOGLEVEL:level}|%{DATA:level}) | (%{DATA:JVMMSG})%{SPACE}| (?<REQ_TIME>%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{TIME}) |%{SPACE}%{GREEDYDATA:MSG}"

Sample Log file :

STATUS | wrapper  | 2023/01/16 08:40:13 | TERM trapped.  Shutting down.
STATUS | wrapper  | 2023/01/16 08:40:27 | <-- Wrapper Stopped
STATUS | wrapper  | 2023/01/16 08:40:28 | WARNING: Could not write lock file /var/lock/subsys: Permission denied
STATUS | wrapper  | 2023/01/16 08:40:28 | --> Wrapper Started as Daemon
STATUS | wrapper  | 2023/01/16 08:40:29 | Launching a JVM...
INFO   | jvm 1    | 2023/01/16 08:40:29 | NOTE: Picked up JDK_JAVA_OPTIONS: --add-reads=java.xml=java.logging --add-exports=java.base/org.apache.karaf.specs.locator=java.xml,ALL-UNNAMED --patch-module java.base=lib/endorsed/org.apache.karaf.specs.locator-%KARAF_VERSION%.jar --patch-module java.xml=lib/endorsed/org.apache.karaf.specs.java.xml-%KARAF_VERSION%.jar --add-opens java.base/java.security=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.naming/javax.naming.spi=ALL-UNNAMED --add-opens java.rmi/sun.rmi.transport.tcp=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.http=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.https=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.jar=ALL-UNNAMED --add-exports=jdk.xml.dom/org.w3c.dom.html=ALL-UNNAMED --add-exports=jdk.naming.rmi/com.sun.jndi.url.rmi=ALL-UNNAMED
INFO   | jvm 1    | 2023/01/16 08:40:29 | WARNING: package org.apache.karaf.specs.locator not in java.base
INFO   | jvm 1    | 2023/01/16 08:40:29 | Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
INFO   | jvm 1    | 2023/01/16 08:40:29 |   Copyright 1999-2006 Tanuki Software, Inc.  All Rights Reserved.
INFO   | jvm 1    | 2023/01/16 08:40:29 |
INFO   | jvm 1    | 2023/01/16 08:58:35 | WARNING: An illegal reflective access operation has occurred
INFO   | jvm 1    | 2023/01/16 08:58:35 | WARNING: Illegal reflective access by org.apache.karaf.shell.support.table.ShellTable (file:/app/data/cache/org.eclipse.osgi/249/0/bundleFile) to field java.io.PrintStream.charOut
INFO   | jvm 1    | 2023/01/16 08:58:35 | WARNING: Please consider reporting this to the maintainers of org.apache.karaf.shell.support.table.ShellTable
INFO   | jvm 1    | 2023/01/16 08:58:35 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
INFO   | jvm 1    | 2023/01/16 08:58:35 | WARNING: All illegal access operations will be denied in a future release
INFO   | jvm 1    | 2023/01/18 16:35:22 | org.apache.felix.resolver.reason.ReasonException: Unable to resolve root: missing requirement [root] osgi.identity; osgi.identity; type=karaf.feature; version="[1.7.37,1.7.37]"; filter:="(&
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at org.apache.felix.resolver.Candidates$MissingRequirementError.toException(Candidates.java:1343)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at org.apache.felix.resolver.ResolverImpl.doResolve(ResolverImpl.java:392)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at org.apache.felix.resolver.ResolverImpl.resolve(ResolverImpl.java:378)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at org.apache.felix.resolver.ResolverImpl.resolve(ResolverImpl.java:332)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at org.apache.karaf.features.internal.region.SubsystemResolver.resolve(SubsystemResolver.java:257)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at org.apache.karaf.features.internal.service.Deployer.deploy(Deployer.java:401)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvision(FeaturesServiceImpl.java:1063)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at org.apache.karaf.features.internal.service.FeaturesServiceImpl.lambda$doProvisionInThread$13(FeaturesServiceImpl.java:998)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at java.base/java.lang.Thread.run(Thread.java:834)
INFO   | jvm 1    | 2023/01/18 16:35:22 | Caused by: org.apache.felix.resolver.reason.ReasonException: Unable to resolve missing requirement osgi.identity; osgi.identity=camel-http4; type=karaf.feature
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at org.apache.felix.resolver.Candidates$MissingRequirementError.toException(Candidates.java:1343)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       ... 12 more
INFO   | jvm 1    | 2023/01/18 16:35:31 | org.apache.felix.resolver.reason.ReasonException: Unable to resolve root: missing requirement [root] osgi.identity; osgi.identity missing requirement ]
INFO   | jvm 1    | 2023/01/18 16:35:31 |       at org.apache.felix.resolver.Candidates$MissingRequirementError.toException(Candidates.java:1343)
INFO   | jvm 1    | 2023/01/18 16:35:31 |       at org.apache.felix.resolver.ResolverImpl.doResolve(ResolverImpl.java:392)
INFO   | jvm 1    | 2023/01/18 16:35:31 |       at org.apache.felix.resolver.ResolverImpl.resolve(ResolverImpl.java:378)
INFO   | jvm 1    | 2023/01/18 16:35:31 |       at org.apache.felix.resolver.ResolverImpl.resolve(ResolverImpl.java:332)
INFO   | jvm 1    | 2023/01/18 16:35:31 |       at org.apache.karaf.features.internal.region.SubsystemResolver.resolve(SubsystemResolver.java:257)
INFO   | jvm 1    | 2023/01/18 16:35:31 |       at org.apache.karaf.features.internal.service.Deployer.deploy(Deployer.java:401)
INFO   | jvm 1    | 2023/01/18 16:35:31 |       at org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvision(FeaturesServiceImpl.java:1063)
INFO   | jvm 1    | 2023/01/18 16:35:31 |       at org.apache.karaf.features.internal.service.FeaturesServiceImpl.lambda$doProvisionInThread$13(FeaturesServiceImpl.java:998)
INFO   | jvm 1    | 2023/01/18 16:35:31 |       at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
INFO   | jvm 1    | 2023/01/18 16:35:31 |       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at java.base/java.lang.Thread.run(Thread.java:834)
        Caused by: org.apache.felix.resolver.reason.ReasonException: Unable to resolve : missing requirement [] osgi.identity;       osgi.identity=camel-http4; type=karaf.feature
     at org.apache.felix.resolver.Candidates$MissingRequirementError.toException(Candidates.java:1343)
     ... 12 more

Refered this article and tried with aggregation but it actuallly not working in this case

Need help for multiline hyperic tomcat wrapper logs with datestamp on each line of the exception

Request: Need help for multiline hyperic tomcat wrapper logs with datestamp on each line of the exception. We have developed below mentioned grok filters for the single line wrapper logs but unable to convert that into multiline as every line has timestamp at the starting of each log line unlike regular tomcat logs. Any help is highly appreciated. Grok Filters Developed so far for single line: JVMNAME %{WORD:jvm} %{INT:number} WRAPPER_DATESTAMP 20%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{HOUR}:?%{MIN…

Need help...

[Badger]

You could use a multiline codec on an input.

 codec => multiline { pattern => "^(|[^\|]+\|[^\|]+\|[^\|]+\|)     " negate => false what => previous auto_flush_interval => 2 }

That assumes lines in the stack trace have multiple spaces, although I would expect them to have a tab. Note that the log level through date are optional, so the third multiline message runs from " Unable to resolve root" at 16:35:31 through "... 12 more".

You can strip out the log level through timestamp in the middle of the [message] field using

mutate { gsub => [ "message", "\n[^\|]+\|[^\|]+\|[^\|]+\|", "" ] }

which results in

   "message" => "INFO   | jvm 1    | 2023/01/18 16:35:31 | org.apache.felix.resolver.reason.ReasonException: Unable to resolve root: missing requirement [root] osgi.identity; osgi.identity missing requirement ]       at org.apache.felix.resolver.Candidates$MissingRequirementError.toException(Candidates.java:1343)       at org.apache.felix.resolver.ResolverImpl.doResolve(ResolverImpl.java:392)       at org.apache.felix.resolver.ResolverImpl.resolve(ResolverImpl.java:378)       at org.apache.felix.resolver.ResolverImpl.resolve(ResolverImpl.java:332)       at org.apache.karaf.features.internal.region.SubsystemResolver.resolve(SubsystemResolver.java:257)       at org.apache.karaf.features.internal.service.Deployer.deploy(Deployer.java:401)       at org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvision(FeaturesServiceImpl.java:1063)       at org.apache.karaf.features.internal.service.FeaturesServiceImpl.lambda$doProvisionInThread$13(FeaturesServiceImpl.java:998)       at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n         at java.base/java.lang.Thread.run(Thread.java:834)\n        Caused by: org.apache.felix.resolver.reason.ReasonException: Unable to resolve : missing requirement [] osgi.identity;       osgi.identity=camel-http4; type=karaf.feature\n     at org.apache.felix.resolver.Candidates$MissingRequirementError.toException(Candidates.java:1343)\n     ... 12 more",

[sai7276p (Sai)]

Thank You Badger