Read logs stored using Enterprise Semantic Logging format

Elastic Stack Logstash
1

We are using Semantic Logging mechanism to log our service request and response in physical files.

Is there a existing GROK Pattern to pull my logs in a good way ?
OR
Any other efficient way to read these logs?

EventId : 1, Level : Informational, Message : Request: 128.194.38.33 GET http://server001/TokenService/api/auth/token, Payload : [id : 4aeeed26-2f5e-4a7b-b4a3-6c35bbad678d] [ipAddress : 168.194.81.321] [method : GET] [requestUri : http://server001/TokenService/api/auth] [content : ] , EventName : RequestInfo, Timestamp : 2016-03-29T02:34:21.7249900Z, ProcessId : 8052, ThreadId : 16104

EventId : 2, Level : Informational, Message : Response: OK, Payload : [id : 4aeeed26-2f5e-4a7b-b4a3-6c35bbad678d] [reasonPhrase : OK] [content : {"Token":"eyJ0eXAiOiJKV12NoZW1hcy54bWxzb2FwLm9yZy93cy8yMDA1LzA1L2lkZW50aXR5L2NsYWltcy9uYW1lIjoiYTExMTM3NzEifQ.YpAKLuFeBtOjm9e4kwNxcdsnJlkd9OH3fXJP2_olzAc","refrToken":"829314e36f933f5f910cf6c"}] , EventName : ResponseInfo, Timestamp : 2016-03-29T02:34:29.5154635Z, ProcessId : 8052, ThreadId : 16104

2

No, but it's not hard to build one using http://grokdebug.herokuapp.com/

3

I tried it. Even had posted this on the forum, but NO reply :frowning:

This is my request log

EventId : 1, Level : Informational, Message : Request, Payload : [sessionID : 43a7be8b-3222-4958-9082-ecc4428f0037] [ipAddress : 172.16.22.88] [method : POST] [requestUri : https://myserver.my,com:4433/MyService/api/click2mob] [content : {"callType":"Sales_Appl","customer":{"firstName":"Service","lastName":"Interceptor","phones":[{"countryCode":"1","areaCode":"612","number":"2221213"}],"emailAddresses":[{"email":"myinterceptor@bestbuy.com"}]}}] , EventName : RequestInfo, Timestamp : 2016-03-22T04:06:01.4157082Z, ProcessId : 9784, ThreadId : 18768

My Current GROK Filter in Logstash -

%{WORD:Event} %{NOTSPACE:Colons} %{WORD:EventID}%{NOTSPACE:Colons} %{WORD:Level} %{NOTSPACE:Colons} %{WORD:LevelDetail}%{NOTSPACE:Colons} %{WORD:Message} %{NOTSPACE:Colons} %{WORD:Request}%{NOTSPACE:Colons} %{WORD:Payload} %{NOTSPACE:Colons} %{NOTSPACE:Colons}%{WORD: Session} %{NOTSPACE:Colons} %{UUID:SessionID}%{NOTSPACE:Colons} %{NOTSPACE:Colons}%{WORD: IPAddress} %{NOTSPACE:Colons} %{IPORHOST:IP}%{NOTSPACE:Colons} %{NOTSPACE:Colons}%{WORD: Method} %{NOTSPACE:Colons} %{WORD:HTTP_VERB}%{NOTSPACE:Colons} %{NOTSPACE:Colons}%{WORD: RequestURI} %{NOTSPACE:Colons} %{URI:URL}%{NOTSPACE:Colons}%{WORD: content} %{NOTSPACE:Colons} %{GREEDYDATA:keyvalues}%{NOTSPACE:Colons} %{WORD: EventName} %{NOTSPACE:Colons} %{WORD:RequestInfo}%{NOTSPACE:Colons} %{WORD:IIS_TimeStamp} %{NOTSPACE:Colons} %{TIMESTAMP_ISO8601:IIS_TimeStamp}%{NOTSPACE:Colons} %{WORD: ProcessID} %{NOTSPACE:Colons} %{WORD:Process_ID}%{NOTSPACE:Colons} %{WORD: ThreadID} %{NOTSPACE:Colons} %{WORD:Thread_ID}

How can I optimize this query to get only the req. values?

Is there anything that can help me here ?

4

I'd start by putting the pattern in http://grokdebug.herokuapp.com/ and the building the pattern piece by piece. You have a lot of NOTSPACE:Colons there, which I assume you are trying to use to represent the :, which is not necessary.

eg EventId : 1, Level : Informational should be %{WORD:Event} : %{NUMBER:EventID}, %{WORD:Level} : %{WORD:EventLevel}.

5

Thanks Mark. That shortens my pattern.

One more thing, in logstash when I add this...it is not captured in ES. Is there anything that I need to add for multiple line search. Since, these logs can span upto multiple lines. And my current logstash conf reads line by line only.

6

You need to look at this then - https://www.elastic.co/guide/en/logstash/current/plugins-filters-multiline.html

7

Will this work for multiline grok pattern. Bcoz it is mentioned for regex only.

8

Ahh sorry I forgot that we moved that to https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html

9

Mark - Can you please help me here. How can I get the entire message. I tried GREEDYDATA, DATA, etc. but couldnt get the entire message value. Is there anything which matches IN BETWEEN -> "Message and EventName : " or a Start and End tag ?

Payload : [sessionID : 7e9c103d-92a8-48f9-a8b9-e1489r7ebeff] [message : Value cannot be null.
Parameter name: phoneObj] [paramName : phoneObj] [source : MySol.API.Common] [stackTrace : at MySol.API.Common.Filters.EmptyParameterFilterAttribute.OnActionExecuting(HttpActionContext actionContext) in e:\Builds\16\IVR\Current.DEV\src\Nightingale\MySol.API.Common\Filters\EmptyParameterFilterAttribute.cs:line 30
** at System.Web.Http.Filters.ActionFilterAttribute.OnActionExecutingAsync(HttpActionContext actionContext, CancellationToken cancellationToken)**
--- End of stack trace from previous location where exception was thrown ---
** at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)**
** at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)**
** at System.Web.Http.Filters.ActionFilterAttribute.d__0.MoveNext()**
--- End of stack trace from previous location where exception was thrown ---
** at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)**
** at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)**
** at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()** ** at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext()** **--- End of stack trace from previous location where exception was thrown ---** ** at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)** ** at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)** ** at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()**
** at System.Web.Http.Controllers.ExceptionFilterResult.d__0.MoveNext()**]
EventName : ResponseInfo

10

Any update ??