Logstash for java server logs

[adg]

Hi
i m having issue while parsing Java Exception stack trace, here we customize our logs
which include some custom parameter just after time stamp

Ex

10:10:03,463#l-21778964801742026### ERROR [PushNotificationServiceImpl] - Error: {}
com.amazonaws.services.sns.model.InvalidParameterException: Invalid parameter: TargetArn Reason: No endpoint found for the target arn specified (Service: AmazonSNS; Status Code: 400; Error Code: InvalidParameter; Request ID: cf2d806b-3621-5d90-a17b-d3251d42c16d)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:889)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:485)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:257)

other type of exception is

10:09:04,508#### ERROR [PushNotificationServiceImpl] - Error: {}
com.amazonaws.services.sns.model.EndpointDisabledException: Endpoint is disabled (Service: AmazonSNS; Status Code: 400; Error Code: EndpointDisabled; Request ID: 752b82b4-1da9-5fab-af50-d1fa3681a6bc)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:889)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:485)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:257)
at com.amazonaws.services.sns.AmazonSNSClient.invoke(AmazonSNSClient.java:2237)
at com.amazonaws.services.sns.AmazonSNSClient.publish(AmazonSNSClient.java:1358)

so as you can see just after TIME there is custom code with in '#l-NUMBER###'
i m unable to parse it not even close of my requirement
and i have to put each thing in septate tag like
"time": 10:09:04,508
"customtransectionid": l-NUMBER
"loglevel":"ERROR"
"classname": "PushNotificationServiceImpl"
"errormessage":"Error: {}"
"exception":"com.amazonaws.services.sns.model.EndpointDisabledException"
"stacktrace":""

to achieve this is use
multiline codec in my input block

codec => multiline {
pattern => "^%{TIME}*"
negate => "true"
what => "previous"
}
in my filter block i use

filter {
grok {
match => {"message"=>"\A%{TIME:timestamp}#%{DATA:alpha}-%{NUMBER:number}###%{SPACE}%{LOGLEVEL:loglevel}%{GREEDYDATA:gdata}"}
}
}

and drop some _grokparsefailure events and change date/timestamp tag to
so please help me so for creating this grok match filer or some other way to do this thing

[Badger]

codec => multiline {
  pattern =>; "^%{TIME}*"
  negate => "true"
  what => "previous"
}

Is this codec part of the input? If so, it cannot reference patterns defined in filters. It might be better to start with a simple stdout { codec => rubydebug } output and make sure that your events are combining the set of lines that you want combined. This document says how to do that with Java stack traces.

Once you can see from the rubydebug that the event has the right set of lines concatenated, you can start trying to either grok or dissect it.

[adg]

it is part of input

input {
file {
path => "/home/dell/app2.log"
start_position => "beginning"
sincedb_path => "/dev/null"
codec => multiline {
pattern => "^%{TIME}*"
negate => "true"
what => "previous"
}
}
}

Filter

filter {
grok {
match => {"message"=>"^%{TIME:time}####%{LOGLEVEL:level} %{GREEDYDATA:gdata}"}
}
date {
match => [ "time" , "yyyy-MM-dd HH:mm:ss.SSS" ]
}

Output

output {
stdout{ codec => rubydebug }
}

[adg]

i m able to out in rubydebug
but i am facing problem to dissect the pattern
i need help in dissecting it
the line u mention has limited , i am unable to find those KEYWORD or example so by looking them i will solve my problem.

[Badger]

Are you able to combine lines in the way you want? Is the problem just grok/dissect? If so, please show us an example of the rubydebug output for an event you want to parse.