Covert the date format to "TIMESTAMP_ISO8601"

Elastic Stack Logstash
1

I wish to parse the tomcat logs. The tomcat logs start with different time formats as follows:

    1.23-Oct-2018 13:19:41.677 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version:        Apache Tomcat/8.5.11
    23-Oct-2018 13:19:41.757 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Jan 10 2017 21:02:52 UTC

2. 2018-10-23 15:00:21.464 http-nio-8085-exec-22 o.h.engine.loading.internal.CollectionLoadContext

3. 15:00:21.464 http-nio-8085-exec-22 o.h.engine.loading.internal.CollectionLoadContext

The first case, I was managed to parsed the logs. But the format is different. In the first case, the time and date parsed as follows :

"date" => "23-Oct-2018"
"time" => "13:19:41.757"

In the second cause, I was able to parse the time and date in "TIMESTAMP_ISO8601" format as follows :

"time_stamp" => "2018-10-23 15:00:21.464"

In the third case, there is no date, I was able to parse the time only :

`   "time" => "15:00:21.464"`

My requirement is to convert all the three to "TIMESTAMP_ISO8601" format. I have tried with "date" filter. But, not getting the required output. Please let us know what are the changes required to be made.

Waiting for the reply.

2

In the third case where there's no date, how is Logstash supposed to know which day it is?

3

Hello,

May I know how "23-Oct-2018 13:19:41.677" can be converted to "TIMESTAMP_ISO8601" format?

In the third case where there's no date, how is Logstash supposed to know which day it is?
Is it possible to append the field "date" from the previous line, if there is no date entry on that particular line.

This the tomcat logs :

1.23-Oct-2018 13:19:50.486 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

2.13:19:51.814 [localhost-startStop-1] DEBUG org.springframework.jndi.JndiTemplate - Looking up JNDI object with name [java:comp/env/logging.exception-conversion-word]

3. 13:19:51.817 [localhost-startStop-1] DEBUG org.springframework.jndi.JndiLocatorDelegate - Converted JNDI name [java:comp/env/logging.exception-conversion-word] not found - trying original name [logging.exception-conversion-word]. javax.naming.NameNotFoundException: Name [logging.exception-conversion-word] is not bound in this Context. Unable to find [logging.exception-conversion-word].

May I know whether the date entry "23-Oct-2018" can be appended to the parsed output for the second and third line.

4

May I know how "23-Oct-2018 13:19:41.677" can be converted to "TIMESTAMP_ISO8601" format?

Use a date filter. The pattern "dd-MMM-yyyy HH:mm:ss.SSS" should do it.

May I know whether the date entry "23-Oct-2018" can be appended to the parsed output for the second and third line.

Maybe an aggregate filter could help? I'm not sure. But really, can't you change that completely idiotic log format?

5

We are unable to change the log format. We are using Tomcat as the application server and this is the tomcat logs.

Here is the sample logs :

1. 23-Oct-2018 13:19:50.486 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

2. 13:19:51.814 [localhost-startStop-1] DEBUG org.springframework.jndi.JndiTemplate - Looking up JNDI object with name [java:comp/env/logging.exception-conversion-word]

3. 13:19:51.817 [localhost-startStop-1] DEBUG org.springframework.jndi.JndiLocatorDelegate - Converted JNDI name [java:comp/env/logging.exception-conversion-word] not found - trying original name [logging.exception-conversion-word]. javax.naming.NameNotFoundException: Name [logging.exception-conversion-word] is not bound in this Context. Unable to find [logging.exception-conversion-word].

In the provided logs, there is date entry only in the first line. I wish to store the date entry in a temporary variable and append that field to the parsed output on the second, third line. The appending should be stopped when a new date entry is found. May I know whether any settings to do that.

6

I was able to copy the value of the field date to another variable mydate using aggregate filter. May I know what is the scope of the variable mydate . Awaiting your reply.

7

Is it possible to convert "2018-11-01T05:30:08.876Z" the value in the field "@timestamp" => 2018-11-01T05:30:08.876Z" to "01-Nov-2018" and copy the value to another field "mydate". May I know what are the steps required to be followed.

Awaiting your reply.

8

I wish to spilt "2018-11-01T05:30:08.876Z" to "2018-11-01" and "05:30:08.876Z. Kindly assist

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.