How to parse the wrapper logs which contains timestamp value at each line

Elastic Stack Logstash
1

I have created grok pattern for single line entries, but I just want to remove the timestamp and extra fields before the java stack trace lines,,, to register a full stack trace to a single field 'MSG'

"(%{LOGLEVEL:level}|%{DATA:level}) | (%{DATA:JVMMSG})%{SPACE}| (?<REQ_TIME>%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{TIME}) |%{SPACE}%{GREEDYDATA:MSG}"

Sample Log file :

STATUS | wrapper  | 2023/01/16 08:40:13 | TERM trapped.  Shutting down.
STATUS | wrapper  | 2023/01/16 08:40:27 | <-- Wrapper Stopped
STATUS | wrapper  | 2023/01/16 08:40:28 | WARNING: Could not write lock file /var/lock/subsys: Permission denied
STATUS | wrapper  | 2023/01/16 08:40:28 | --> Wrapper Started as Daemon
STATUS | wrapper  | 2023/01/16 08:40:29 | Launching a JVM...
INFO   | jvm 1    | 2023/01/16 08:40:29 | NOTE: Picked up JDK_JAVA_OPTIONS: --add-reads=java.xml=java.logging --add-exports=java.base/org.apache.karaf.specs.locator=java.xml,ALL-UNNAMED --patch-module java.base=lib/endorsed/org.apache.karaf.specs.locator-%KARAF_VERSION%.jar --patch-module java.xml=lib/endorsed/org.apache.karaf.specs.java.xml-%KARAF_VERSION%.jar --add-opens java.base/java.security=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.naming/javax.naming.spi=ALL-UNNAMED --add-opens java.rmi/sun.rmi.transport.tcp=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.http=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.https=ALL-UNNAMED --add-exports=java.base/sun.net.www.protocol.jar=ALL-UNNAMED --add-exports=jdk.xml.dom/org.w3c.dom.html=ALL-UNNAMED --add-exports=jdk.naming.rmi/com.sun.jndi.url.rmi=ALL-UNNAMED
INFO   | jvm 1    | 2023/01/16 08:40:29 | WARNING: package org.apache.karaf.specs.locator not in java.base
INFO   | jvm 1    | 2023/01/16 08:40:29 | Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
INFO   | jvm 1    | 2023/01/16 08:40:29 |   Copyright 1999-2006 Tanuki Software, Inc.  All Rights Reserved.
INFO   | jvm 1    | 2023/01/16 08:40:29 |
INFO   | jvm 1    | 2023/01/16 08:58:35 | WARNING: An illegal reflective access operation has occurred
INFO   | jvm 1    | 2023/01/16 08:58:35 | WARNING: Illegal reflective access by org.apache.karaf.shell.support.table.ShellTable (file:/app/data/cache/org.eclipse.osgi/249/0/bundleFile) to field java.io.PrintStream.charOut
INFO   | jvm 1    | 2023/01/16 08:58:35 | WARNING: Please consider reporting this to the maintainers of org.apache.karaf.shell.support.table.ShellTable
INFO   | jvm 1    | 2023/01/16 08:58:35 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
INFO   | jvm 1    | 2023/01/16 08:58:35 | WARNING: All illegal access operations will be denied in a future release
INFO   | jvm 1    | 2023/01/18 16:35:22 | org.apache.felix.resolver.reason.ReasonException: Unable to resolve root: missing requirement [root] osgi.identity; osgi.identity; type=karaf.feature; version="[1.7.37,1.7.37]"; filter:="(&
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at org.apache.felix.resolver.Candidates$MissingRequirementError.toException(Candidates.java:1343)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at org.apache.felix.resolver.ResolverImpl.doResolve(ResolverImpl.java:392)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at org.apache.felix.resolver.ResolverImpl.resolve(ResolverImpl.java:378)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at org.apache.felix.resolver.ResolverImpl.resolve(ResolverImpl.java:332)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at org.apache.karaf.features.internal.region.SubsystemResolver.resolve(SubsystemResolver.java:257)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at org.apache.karaf.features.internal.service.Deployer.deploy(Deployer.java:401)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvision(FeaturesServiceImpl.java:1063)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at org.apache.karaf.features.internal.service.FeaturesServiceImpl.lambda$doProvisionInThread$13(FeaturesServiceImpl.java:998)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at java.base/java.lang.Thread.run(Thread.java:834)
INFO   | jvm 1    | 2023/01/18 16:35:22 | Caused by: org.apache.felix.resolver.reason.ReasonException: Unable to resolve missing requirement osgi.identity; osgi.identity=camel-http4; type=karaf.feature
INFO   | jvm 1    | 2023/01/18 16:35:22 |       at org.apache.felix.resolver.Candidates$MissingRequirementError.toException(Candidates.java:1343)
INFO   | jvm 1    | 2023/01/18 16:35:22 |       ... 12 more
INFO   | jvm 1    | 2023/01/18 16:35:31 | org.apache.felix.resolver.reason.ReasonException: Unable to resolve root: missing requirement [root] osgi.identity; osgi.identity missing requirement ]
INFO   | jvm 1    | 2023/01/18 16:35:31 |       at org.apache.felix.resolver.Candidates$MissingRequirementError.toException(Candidates.java:1343)
INFO   | jvm 1    | 2023/01/18 16:35:31 |       at org.apache.felix.resolver.ResolverImpl.doResolve(ResolverImpl.java:392)
INFO   | jvm 1    | 2023/01/18 16:35:31 |       at org.apache.felix.resolver.ResolverImpl.resolve(ResolverImpl.java:378)
INFO   | jvm 1    | 2023/01/18 16:35:31 |       at org.apache.felix.resolver.ResolverImpl.resolve(ResolverImpl.java:332)
INFO   | jvm 1    | 2023/01/18 16:35:31 |       at org.apache.karaf.features.internal.region.SubsystemResolver.resolve(SubsystemResolver.java:257)
INFO   | jvm 1    | 2023/01/18 16:35:31 |       at org.apache.karaf.features.internal.service.Deployer.deploy(Deployer.java:401)
INFO   | jvm 1    | 2023/01/18 16:35:31 |       at org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvision(FeaturesServiceImpl.java:1063)
INFO   | jvm 1    | 2023/01/18 16:35:31 |       at org.apache.karaf.features.internal.service.FeaturesServiceImpl.lambda$doProvisionInThread$13(FeaturesServiceImpl.java:998)
INFO   | jvm 1    | 2023/01/18 16:35:31 |       at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
INFO   | jvm 1    | 2023/01/18 16:35:31 |       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at java.base/java.lang.Thread.run(Thread.java:834)
        Caused by: org.apache.felix.resolver.reason.ReasonException: Unable to resolve : missing requirement [] osgi.identity;       osgi.identity=camel-http4; type=karaf.feature
     at org.apache.felix.resolver.Candidates$MissingRequirementError.toException(Candidates.java:1343)
     ... 12 more

Refered this article and tried with aggregation but it actuallly not working in this case

Need help...

2

You could use a multiline codec on an input.

 codec => multiline { pattern => "^(|[^\|]+\|[^\|]+\|[^\|]+\|)     " negate => false what => previous auto_flush_interval => 2 }

That assumes lines in the stack trace have multiple spaces, although I would expect them to have a tab. Note that the log level through date are optional, so the third multiline message runs from " Unable to resolve root" at 16:35:31 through "... 12 more".

You can strip out the log level through timestamp in the middle of the [message] field using

mutate { gsub => [ "message", "\n[^\|]+\|[^\|]+\|[^\|]+\|", "" ] }

which results in

   "message" => "INFO   | jvm 1    | 2023/01/18 16:35:31 | org.apache.felix.resolver.reason.ReasonException: Unable to resolve root: missing requirement [root] osgi.identity; osgi.identity missing requirement ]       at org.apache.felix.resolver.Candidates$MissingRequirementError.toException(Candidates.java:1343)       at org.apache.felix.resolver.ResolverImpl.doResolve(ResolverImpl.java:392)       at org.apache.felix.resolver.ResolverImpl.resolve(ResolverImpl.java:378)       at org.apache.felix.resolver.ResolverImpl.resolve(ResolverImpl.java:332)       at org.apache.karaf.features.internal.region.SubsystemResolver.resolve(SubsystemResolver.java:257)       at org.apache.karaf.features.internal.service.Deployer.deploy(Deployer.java:401)       at org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvision(FeaturesServiceImpl.java:1063)       at org.apache.karaf.features.internal.service.FeaturesServiceImpl.lambda$doProvisionInThread$13(FeaturesServiceImpl.java:998)       at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n         at java.base/java.lang.Thread.run(Thread.java:834)\n        Caused by: org.apache.felix.resolver.reason.ReasonException: Unable to resolve : missing requirement [] osgi.identity;       osgi.identity=camel-http4; type=karaf.feature\n     at org.apache.felix.resolver.Candidates$MissingRequirementError.toException(Candidates.java:1343)\n     ... 12 more",
3

Thank You Badger :pray:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.