Any Help Un-structure log message to map structure message in Logstash

Elastic Stack Logstash
1

Hello, I have the log message
2015-11-18 21:11:38,693 [WARN ] [xx.web.common.filter.RequestFilter] NDC[UserPrincipal(ABCDF22602)] request (/member/control/loginAction) exceeded threshold; elapsed milliseconds since start: 188814

Can any one help me. How to tag or map the each same stored/moved to elasticsearch or any output resource

dose GROK plugin help for this mapping? any other plugin available to map/associate the each value to some TAG
I am trying like this in grok plugin but not working

match => { "timestamp" => "%{TOMCAT_DATESTAMP:timestamp}"}
match => { "level" => "[%{LOGLEVEL:level}]" }
match => { "class" => "[%{JAVACLASS:class}]" }
match => { "logmessage" => "%{JAVALOGMESSAGE:logmessage}" }

2

Yes, use the grok filter but correctly. It should look similar to this:

grok {
  match => {
    "message" => "%{TOMCAT_DATESTAMP:timestamp} \[%{LOGLEVEL:level}\] ..."
  }
}

(Note the escaping of the square brackets.)

3

Hi Magnus,

can we map this NDC[UserPrincipal(ABCDEF22602) with Tag, Subtags Like below, Please give me syntax

MainTag : NDC[SubTag: UserPrincipal(UserTag: ABCDEF22602)

4

Hi Magnus,

I have tried as per your suggestion but its not mapping... I am seeing below output

{
"message" => "2015-12-01 23:58:00,330 [INFO ] [xx.cbm.framework.engine.actionFrame.ActionFrameMonitor] NDC[] Completed checking
for expired ActionFrames (there are now 0 ActionFrame references being monitored).\r",
"@version" => "1",
"@timestamp" => "2015-12-03T09:38:37.097Z",
"host" => "MYVDI-XXX",
"path" => "D:\basefarm\logs\myserver1.log",
"tags" => [
[0] "_grokparsefailure"
]
}

I am expecting below response

{
"message" => "2015-12-01 23:58:00,330 [INFO ] [xx.cbm.framework.engine.actionFrame.ActionFrameMonitor] NDC[UserPrincipal(abcdef12121)] Completed checking
for expired ActionFrames (there are now 0 ActionFrame references being monitored).\r",
"timestamp" => ""2015-12-01 23:58:00",
"level" => "INFO",
"javaclass" => "xx.bac.framework.engine.actionFrame.ActionFrameMonitor",
"userid" => "abcdef12121",
"logmessage" => "Completed checking for expired ActionFrames (there are now 0 ActionFrame references being monitored).\r"
"@version" => "1",
"@timestamp" => "2015-12-03T09:38:37.097Z",
"host" => "MYVDI-2033",
"path" => "D:\basefarm\logs\myserver1.log",
"tags" => [
[0] "_grokparsefailure"
]
}

5

Without seeing your configuration it's impossible to know what's wrong. I suggest you use http://grokconstructor.appspot.com/ as a help to create a grok expression that matches your data.

6

Please see my input configurations

input {
file {
path => "D:\basefarm\logs\myserver1.log"
start_position => "beginning"
}
}

filter {
grok {
match => { "message" => "%{TOMCAT_DATESTAMP:timestamp} [%{LOGLEVEL:level}] [%{JAVACLASS:class}] %{JAVALOGMESSAGE:logmessage}"}
}
grok{
match => { "exceptions" => "%{JAVASTACKTRACEPART}" }
}
date {
match => [ "timestamp" , "yyyy-mm-dd HH:mm:ss,SSS Z" ]
}
}

output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
}
stdout { codec => rubydebug }
}

Please help me now

7

One immediate problem is that you're not escaping the square brackets as I instructed you to do in a previous post. If it still doesn't work, use http://grokconstructor.appspot.com/. Over and out.

8

no it is editor issue not showing single escape "" charactor in preview or in posted message i have added escaping the square brackets as you instructed..

9

Hi Magnus,

I have checked grokconstructor but not able to find or convert the log message to required pattern maching please help me for the below log message

2015-11-18 21:11:38,693 [WARN ] [xx.web.common.filter.RequestFilter] NDC[UserPrincipal(ABCDF22602)] request (/member/control/loginAction) exceeded threshold; elapsed milliseconds since start: 188814

This is not working please help me
{ "message" => "%{TOMCAT_DATESTAMP:timestamp} \[%{LOGLEVEL:level}\] \[%{JAVACLASS:class}\] %{JAVALOGMESSAGE:logmessage}"}

This is the log4j pattern we using %d [%-5p] [%c] NDC[%x] %m%n

10

To debug this, start with the simplest possible expression:

%{TOMCAT_DATESTAMP:timestamp}.*

Does this work? If yes, add the next part:

%{TOMCAT_DATESTAMP:timestamp} \[%{LOGLEVEL:level}\].*

And so on. When it stops working you have found the problematic part of the expression.

Looking at the definition of TOMCAT_DATESTAMP,

it ends with ISO8601_TIMEZONE. There's no timezone in your log. Replacing TOMCAT_DATESTAMP with TIMESTAMP_ISO8601 could help.

11

Thank you Magnus,

I have a question, I have give grok filter with following expression for message... in each event there may be no track trace still this is valid exprestion?
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:logtimestamp} \[%{LOGLEVEL:level}\ ] \[%{JAVACLASS:class}\] %{JAVALOGMESSAGE:logmessage} "%{JAVASTACKTRACEPART:exceptions}"}
}

one more question if log message contains curling brackets { } the message is not parsing... grok filter failing to parse... is there any thing we need to add in grok filter to parse the message which contain { }

12

I have a question, I have give grok filter with following expression for message... in each event there may be no track trace still this is valid exprestion?

grok {
match => { "message" => "%{TIMESTAMP_ISO8601:logtimestamp} [%{LOGLEVEL:level}\ ] [%{JAVACLASS:class}] %{JAVALOGMESSAGE:logmessage} "%{JAVASTACKTRACEPART:exceptions}"}
}

There's an extra double quote just before the JAVASTACKTRACEPART token so Logstash won't accept it as it stands.

Given the definition of JAVASTACKTRACEPART,

it's clear that there are no optional elements, i.e. a grok expression that ends with a reference to JAVASTACKTRACEPART really must end with a stacktrace. You could make it optional by changing the end of your expression like this (note addition of parenthesis and question mark):

... %{JAVALOGMESSAGE:logmessage}( %{JAVASTACKTRACEPART:exceptions})?