Can someone give pattern for Tomcat logs

Elastic Stack Logstash
1

Hi,

I am using logstash to process Tomcat logs and i am using below pattern :smile: Java Logs
JAVATHREAD (?:[A-Z]{2}-Processor[\d]+)
JAVACLASS (?:[a-zA-Z0-9-]+.)+[A-Za-z0-9$]+
JAVAFILE (?:[A-Za-z0-9_.-]+)
JAVASTACKTRACEPART at %{JAVACLASS:class}.%{WORD:method}(%{JAVAFILE:file}:%{NUMBER:line})
JAVALOGMESSAGE (.*)

MMM dd, yyyy HH:mm:ss eg: Jan 9, 2014 7:13:13 AM

CATALINA_DATESTAMP %{MONTH} %{MONTHDAY}, 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) (?:AM|PM)

yyyy-MM-dd HH:mm:ss,SSS ZZZ eg: 2014-01-09 17:32:25,527 -0800

TOMCAT_DATESTAMP 20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) %{ISO8601_TIMEZONE}
CATALINALOG %{CATALINA_DATESTAMP:timestamp} %{JAVACLASS:class} %{JAVALOGMESSAGE:logmessage}

2014-01-09 20:03:28,269 -0800 | ERROR | com.example.service.ExampleService - something compeletely unexpected happened...

TOMCATLOG %{TOMCAT_DATESTAMP:timestamp} | %{LOGLEVEL:level} | %{JAVACLASS:class} - %{JAVALOGMESSAGE:logmessage}

and my conf file looks like below
input { stdin { } }

filter {
multiline {
patterns_dir => ["/www/elastic/logstash-1.4.2/conf/pattern_java"]
pattern => "(^%{TOMCAT_DATESTAMP})|(^%{CATALINA_DATESTAMP})"
negate => true
what => "previous"
}
grok {
patterns_dir => ["/www/elastic/logstash-1.4.2/conf/pattern_java"]
match => [ "message", "%{TOMCATLOG}", "message", "%{CATALINALOG}" ]
}
date {
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss,SSS Z", "MMM dd, yyyy HH:mm:ss a" ]
}
}
output {
elasticsearch { host => poc1 protocol => http }
stdout { codec => rubydebug }
}

Running logstash : ./logstash -f ../conf/logstash_tomcat.conf

Giving logs to terminal :

2014-01-09 17:32:25,527 -0800 | ERROR | com.example.controller.ApiController - Request exception
javax.xml.ws.WebServiceException: Failed to access the WSDL at: https://api.example.com/DataServices/Data?WSDL. It failed with:
Connection reset.
at com.example.webservices.Data.(Data.java:50)
at com.example.service.soap.DataService.submitRequest(DataService.groovy:28)
at com.example.service.request.RequestService.addRequest(RequestService.groovy:26)
at com.example.controller.ApiController.request(ApiController.groovy:692)
at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:200)
at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:311)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:776)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:705)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
Caused by: java.net.SocketException: Connection reset
... 17 more

Its not processing. Please suggest.

Regards,
Raman

2

Are you feeding Logstash more than one message? If not, Logstash is waiting for a line that starts with a timestamp so that it can flush the current message. Try feeding it another message and see if you get the first message back in parsed form.

(Hint: If you have code-like text that shouldn't be formatted with Markdown, like configuration files and logs snippets, select the text and press Ctrl+K.)

3

Thanks for the response.

I am using one message at a time and waiting. Can you please check my config file if it is correct ?

Regards,
Raman

4

Your configuration might be fine, but Logstash won't emit the first message until it sees the start of the second message. So, feed it two messages.

5

Hi Magnus,

I have tried and configured it to accept apache logs . Apache logs are coming but the same problem, its failing to give any output for Tomcat logs

6

When i remove multiline filter plugin then its giving me result with grokfailure.
Please suggest

7

With debug mode its giving me below error :smile:

8

Failed parsing date from field {:field=>"timestamp", :value=>"Tue Jun 02 11:30:46 2015", :exception=>java.lang.IllegalArgumentException: Invalid format: "Tue Jun 02 11:30:46 2015", :level=>:warn}

9

First things first. Focus on the multiline problem. Did you try my suggestion?

10

Yes, i tried to feed two and more than two messages.Still its not giving response.

11

Okay. Probably something wrong with your patterns then. You can use grok to make sure that CATALINA_DATESTAMP and TOMCAT_DATESTAMP match correctly.

12

My logs are like these given below

[2015-03-23 13:57:16,866 ] INFO AprLifecycleListener -- OpenSSL successfully initialized (OpenSSL0.9.8e-fips-rhel5 01 Jul 2008)
[2015-03-23 13:57:17,719 ] INFO AbstractProtocol -- Initializing ProtocolHandler ["http-bio--8443"]
[2015-03-23 13:57:18,276 ] ERROR AbstractProtocol -- Failed to initialize end point associated with ProtocolHandler ["http-bio--8443"]
java.net.BindException: Address already in use /:8443
at org.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:406)
at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:610)
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:429)
at org.apache.coyote.http11.AbstractHttp11JsseProtocol.init(AbstractHttp11JsseProtocol.java:119)

I am using below patterns for above logs

CATALINA_DATESTAMP %{MONTH} %{MONTHDAY}, 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) (?:AM|PM)
TOMCAT_DATESTAMP 20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) %{ISO8601_TIMEZONE}

CATALINALOG %{CATALINA_DATESTAMP:timestamp} %{JAVACLASS:class} %{JAVALOGMESSAGE:logmessage}
TOMCATLOG %{TOMCAT_DATESTAMP:timestamp} | %{LOGLEVEL:level} | %{JAVACLASS:class} - %{JAVALOGMESSAGE:logmessage}

Please suggest

13

You're not taking the square brackets around the timestamp into consideration and you only have a single hyphen between the logger name and the log message. This should work better. If it doesn't work you'll have to dig into it yourself or hope that someone else can help.

TOMCATLOG \[%{TOMCAT_DATESTAMP:timestamp}\] \| %{LOGLEVEL:level} \| %{JAVACLASS:class} -- %{JAVALOGMESSAGE:logmessage}
14

Thanks for your help. Still no luck. I'll check it and update you if i get something.

15

Raman_Saini, did it work, at last?

16

HI ,

I am trying to parse below line ,

2017-01-23 00:00:00,234 INFO [FTP_Channel-0] aps.pistribution (PhannelImpl.java:205) - FTP_Channel-0Refill completed>

Below is my grok pattern :

20%{YEAR}-%{MONTHNUM}-%{MONTHDAY}%{HOUR}:?%{MINUTE}(?::?%{SECOND})%{SPACE}%{LOGLEVEL:level}%{SPACE}[%{GREEDYDATA:Channel}]%{SPACE}%{WORD:Project}%{SPACE}%{JAVACLASS:class}%{SPACE}%{GREEDYDATA:logmsg}

Can you please help me with what am missing here

17

@pDonS, please start a new thread/topic for your unrelated question.