Getting started with ELK catalina.out login

Elastic Stack Elasticsearch
#1

Hey there, i'm getting started with elk solution my catalina.out is login in the follow format:
"Mar 23, 2016 11:33:14 AM" .

I parse the log using grok as follow
%{MONTH} +%{MONTHDAY}, %{YEAR} %{TIME} %{WORD} %{JAVACLASS} %{WORD:message}\n%{LOGLEVEL:loglevel}: %{DATA}\n%{DATA}\n

Now i'm creating a index on elastic with mapping like this:
"mappings" : {
"default" : {
"properties" : {
"timestamp": {
"type": "date",
"format": "MMM dd, YYYY hh:mm:ss"
},
"JavaClass": {
"type": "string","index":"not_analyzed"
},
"methodName":{
"type":"string","index":"not_analyzed"
},
"message": {
"type": "string"
},
"loglevel": {
"type": "string"
},
"extradata": {
"type": "string"
}
}
}
}
But in kibana i have the follow message
"Discover: An error occurred with your request. Reset your inputs and try again."

I'm missing something?

If anyone is able to assist me please let me know

Thank's,
Cristian

#2

Is there anything else in the KB logs themselves?

#3

Thank for the replay

Yes it's i'm close to solve this. There is a way to say to grok take everything esle for that event and parse as GREEDYDATA for example?

Thanks again,
Cristian

#4

Yes it's i'm close to solve this. There is a way to say to grok take everything esle for that event and parse as GREEDYDATA for example?

Yes, just put %{GREEDYDATA:whatever} to have it capture as much as it can. Without specifics of what you're trying to do it's impossible to be specific.

#5

Thanks for replay Magnus,

I have a log like this:

May 19, 2016 7:15:42 PM com.sun.xml.ws.transport.http.HttpAdapter fixQuotesAroundSoapAction
INFO: Received WS-I BP non-conformant Unquoted SoapAction HTTP header:
sendResponseStatus 530 SPA not supported by the object
java.lang.Throwable

When i try the grok debug all work just fine but when i execute the same code in logstash i have a grokparsefailure.

Im using the following custom pattern

CATALINA_DATESTAMP %{MONTH} %{MONTHDAY}, 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) (?:AM|PM)
CATALINALOG %{CATALINA_DATESTAMP:timestamp} %{JAVACLASS:class} %{JAVALOGMESSAGE}\c\n%{LOGLEVEL: logLevel}:
JAVALOGMESSAGE (.*)

When i was testing that in logstash i realize that the \n or \c\n is not working in my logstash. Any suggestions? I have installed Logstash 2.3.1 All Plugins

Thanks.

#6

Have you used the multiline codec to join the lines of the log so that the text you're actually parsing contains the whole message? Please use a stdout { codec => rubydebug } output to debug this matter, and show the output of that so that we can see exactly what happens.

#7

Yes i did. Here is my whole filter

grok {
    match => [ "message", "%{CATALINALOG}" ]

}
multiline {
pattern => "(^%{CATALINA_DATESTAMP})"
negate => true
what => "previous"
}
}

I'm testing that like you saying already, that's my output:

   "message" => "May 19, 2016 7:15:42 PM com.sun.xml.ws.transport.http.HttpAdapter fixQuotesAroundSoapAction\nINFO: Received WS-I BP non-conformant Unquoted SoapAction HTTP header: \nsendResponseStatus 530 SPA not supported by the object",
  "@version" => "1",
"@timestamp" => "2016-05-31T19:29:15.765Z",
      "host" => "FBDtest",
      "tags" => [
    [0] "_grokparsefailure",
    [1] "multiline"
]

}

I saw a lot of examples doing the same but seems i'm the only one who have that problem :sweat:

#8

Start by establishing what part of your expression that doesn't match. Start with the simplest possible expression, %{MONTH} in your case, and add token after token until it stops working.

What's up with the \c? I don't believe that's a standard escape sequence, and it's obviously not part of your message.

#9

I already know which part don't match but that only happens on logstash.

I already try skiping the \c\n%{LOGLEVEL: logLevel}: part and it work fine
and even try with just \n%{LOGLEVEL: logLevel}:
Same output

In the grok debugger work in both cases but not in my logstash. We are going anywhere here, thanks for reply's i'll keep trying to solve that.

Regards,
Cristian

#10