Read logs stored using Enterprise Semantic Logging format

We are using Semantic Logging mechanism to log our service request and response in physical files.

Is there a existing GROK Pattern to pull my logs in a good way ?
OR
Any other efficient way to read these logs?

EventId : 1, Level : Informational, Message : Request: 128.194.38.33 GET http://server001/TokenService/api/auth/token, Payload : [id : 4aeeed26-2f5e-4a7b-b4a3-6c35bbad678d] [ipAddress : 168.194.81.321] [method : GET] [requestUri : http://server001/TokenService/api/auth] [content : ] , EventName : RequestInfo, Timestamp : 2016-03-29T02:34:21.7249900Z, ProcessId : 8052, ThreadId : 16104

EventId : 2, Level : Informational, Message : Response: OK, Payload : [id : 4aeeed26-2f5e-4a7b-b4a3-6c35bbad678d] [reasonPhrase : OK] [content : {"Token":"eyJ0eXAiOiJKV12NoZW1hcy54bWxzb2FwLm9yZy93cy8yMDA1LzA1L2lkZW50aXR5L2NsYWltcy9uYW1lIjoiYTExMTM3NzEifQ.YpAKLuFeBtOjm9e4kwNxcdsnJlkd9OH3fXJP2_olzAc","refrToken":"829314e36f933f5f910cf6c"}] , EventName : ResponseInfo, Timestamp : 2016-03-29T02:34:29.5154635Z, ProcessId : 8052, ThreadId : 16104

No, but it's not hard to build one using http://grokdebug.herokuapp.com/

I tried it. Even had posted this on the forum, but NO reply :frowning:

This is my request log

EventId : 1, Level : Informational, Message : Request, Payload : [sessionID : 43a7be8b-3222-4958-9082-ecc4428f0037] [ipAddress : 172.16.22.88] [method : POST] [requestUri : https://myserver.my,com:4433/MyService/api/click2mob] [content : {"callType":"Sales_Appl","customer":{"firstName":"Service","lastName":"Interceptor","phones":[{"countryCode":"1","areaCode":"612","number":"2221213"}],"emailAddresses":[{"email":"myinterceptor@bestbuy.com"}]}}] , EventName : RequestInfo, Timestamp : 2016-03-22T04:06:01.4157082Z, ProcessId : 9784, ThreadId : 18768

My Current GROK Filter in Logstash -

%{WORD:Event} %{NOTSPACE:Colons} %{WORD:EventID}%{NOTSPACE:Colons} %{WORD:Level} %{NOTSPACE:Colons} %{WORD:LevelDetail}%{NOTSPACE:Colons} %{WORD:Message} %{NOTSPACE:Colons} %{WORD:Request}%{NOTSPACE:Colons} %{WORD:Payload} %{NOTSPACE:Colons} %{NOTSPACE:Colons}%{WORD: Session} %{NOTSPACE:Colons} %{UUID:SessionID}%{NOTSPACE:Colons} %{NOTSPACE:Colons}%{WORD: IPAddress} %{NOTSPACE:Colons} %{IPORHOST:IP}%{NOTSPACE:Colons} %{NOTSPACE:Colons}%{WORD: Method} %{NOTSPACE:Colons} %{WORD:HTTP_VERB}%{NOTSPACE:Colons} %{NOTSPACE:Colons}%{WORD: RequestURI} %{NOTSPACE:Colons} %{URI:URL}%{NOTSPACE:Colons}%{WORD: content} %{NOTSPACE:Colons} %{GREEDYDATA:keyvalues}%{NOTSPACE:Colons} %{WORD: EventName} %{NOTSPACE:Colons} %{WORD:RequestInfo}%{NOTSPACE:Colons} %{WORD:IIS_TimeStamp} %{NOTSPACE:Colons} %{TIMESTAMP_ISO8601:IIS_TimeStamp}%{NOTSPACE:Colons} %{WORD: ProcessID} %{NOTSPACE:Colons} %{WORD:Process_ID}%{NOTSPACE:Colons} %{WORD: ThreadID} %{NOTSPACE:Colons} %{WORD:Thread_ID}

How can I optimize this query to get only the req. values?

Is there anything that can help me here ?

I'd start by putting the pattern in http://grokdebug.herokuapp.com/ and the building the pattern piece by piece. You have a lot of NOTSPACE:Colons there, which I assume you are trying to use to represent the :, which is not necessary.

eg EventId : 1, Level : Informational should be %{WORD:Event} : %{NUMBER:EventID}, %{WORD:Level} : %{WORD:EventLevel}.

Thanks Mark. That shortens my pattern.

One more thing, in logstash when I add this...it is not captured in ES. Is there anything that I need to add for multiple line search. Since, these logs can span upto multiple lines. And my current logstash conf reads line by line only.

You need to look at this then - https://www.elastic.co/guide/en/logstash/current/plugins-filters-multiline.html

Will this work for multiline grok pattern. Bcoz it is mentioned for regex only.

Ahh sorry I forgot that we moved that to https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html

Mark - Can you please help me here. How can I get the entire message. I tried GREEDYDATA, DATA, etc. but couldnt get the entire message value. Is there anything which matches IN BETWEEN -> "Message and EventName : " or a Start and End tag ?

Payload : [sessionID : 7e9c103d-92a8-48f9-a8b9-e1489r7ebeff] [message : Value cannot be null.
Parameter name: phoneObj] [paramName : phoneObj] [source : MySol.API.Common] [stackTrace : at MySol.API.Common.Filters.EmptyParameterFilterAttribute.OnActionExecuting(HttpActionContext actionContext) in e:\Builds\16\IVR\Current.DEV\src\Nightingale\MySol.API.Common\Filters\EmptyParameterFilterAttribute.cs:line 30
** at System.Web.Http.Filters.ActionFilterAttribute.OnActionExecutingAsync(HttpActionContext actionContext, CancellationToken cancellationToken)**
--- End of stack trace from previous location where exception was thrown ---
** at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)**
** at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)**
** at System.Web.Http.Filters.ActionFilterAttribute.d__0.MoveNext()**
--- End of stack trace from previous location where exception was thrown ---
** at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)**
** at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)**
** at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()** ** at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext()** **--- End of stack trace from previous location where exception was thrown ---** ** at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)** ** at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)** ** at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()**
** at System.Web.Http.Controllers.ExceptionFilterResult.d__0.MoveNext()**]
EventName : ResponseInfo

Any update ??