Parsing multiline unstructured log..Please give grok for this below log\

Elastic Stack Logstash
1

INFO 2019-04-23 01:57:00,258 [AsyncHttpClient-Callback] org.mule.runtime.core.internal.util.queue.QueueXaResourceManager: Starting ResourceManager
INFO 2019-04-23 01:57:00,258 [AsyncHttpClient-Callback] org.mule.runtime.core.internal.util.queue.QueueXaResourceManager: Started ResourceManager
INFO 2019-04-23 01:57:00,259 [AsyncHttpClient-Callback] org.mule.runtime.core.internal.construct.FlowConstructLifecycleManager: Starting flow: request-routerFlow-Sapi
INFO 2019-04-23 01:57:00,299 [AsyncHttpClient-Callback] org.mule.runtime.core.internal.construct.FlowConstructLifecycleManager: Starting flow: custom-transactionid-Flow
INFO 2019-04-23 01:57:00,323 [AsyncHttpClient-Callback] org.mule.runtime.core.internal.construct.FlowConstructLifecycleManager: Starting flow: hannaford-togo-xapi-main
INFO 2019-04-23 01:57:00,353 [AsyncHttpClient-Callback] org.mule.runtime.core.internal.construct.FlowConstructLifecycleManager: Starting flow: hannaford-togo-xapi-console
INFO 2019-04-23 01:57:00,361 [AsyncHttpClient-Callback] org.mule.runtime.core.internal.construct.FlowConstructLifecycleManager: Starting flow: post:\qbusterFTE:application\json:hannaford-togo-xapi-config
INFO 2019-04-23 01:57:00,385 [AsyncHttpClient-Callback] org.mule.runtime.core.internal.context.DefaultMuleContext:

  • Application: hannaford-togo-xapi-group2 *
  • OS encoding: UTF-8, Mule encoding: UTF-8 *
  •                                                                *
2

That log file looks pretty structured. Have a look at http://grokdebug.herokuapp.com/ or Kibana's inbuilt debugger (Dev Tools -> Grok Debugger) to try out your grok filters

I can see from the logs that you can use LOGLEVEL and TIMESTAMP_ISO8601 patterns (http://grokdebug.herokuapp.com/patterns). In your beats configuration, you can tell it that the log is multiline and split new log records when a line hits LOGLEVEL

3

Thank you for the reply
But Here i have the file which contains some lines which are as below and i really need Help in parsing this multi line pattern..TIA

4

INFO 2019-05-03 13:22:16,015 [[MuleRuntime].cpuLight.03: [hford-togo-xapi-group2].hford-togo-xapi-main.CPU_LITE @48cef31a] 0-ff240161-6dc7-11e9-bcae-0242ac120002org.mule.runtime.core.internal.processor.LoggerMessageProcessor: {
"QBusterHeader": {
"LineNo": 1,
"StartTime": 142825,
"EndTime": 0,
"NumberOfItems": 5,
"UserName": "HTG-8345",
"ReferenceNumber": "5019406"
},
"QBusterDetail": [{
"LineNo": 2,
"PluID": 5923,
"Qty": 1,
"RandomWeight_FG": "N"
}, {
"LineNo": 3,
"PluID": 4460030768,
"Qty": 1,
"RandomWeight_FG": "N"
}, {
"LineNo": 4,
"PluID": 4460030768,
"Qty": 1,
"RandomWeight_FG": "N"
}, {
"LineNo": 5,
"PluID": 2100065883,
"Qty": 1,
"RandomWeight_FG": "N"
}, {
"LineNo": 6,
"PluID": 5480002001,
"Qty": 1,
"RandomWeight_FG": "N"
}
]
}
org.mule.runtime.core.internal.message.DefaultMessageBuilder$MessageImplementation
{
payload=org.mule.runtime.core.internal.streaming.bytes.ManagedCursorStreamProvider$ManagedCursorDecorator@65d5ed8e
mediaType=application/json; charset=UTF-8
attributes=org.mule.extension.http.api.HttpRequestAttributes
{
Request path=/htg/v1/qbusterFTE
Raw request path=/htg/v1/qbusterFTE
Method=POST
Listener path=/htg/v1/*
Local Address=172.18.0.2:8081
Query String=
Relative Path=/htg/v1/qbusterFTE
Masked Request Path=/qbusterFTE
Remote Address=172.18.0.1:58994
Request Uri=/htg/v1/qbusterFTE
Raw request Uri=/htg/v1/qbusterFTE
Scheme=http
Version=HTTP/1.1
Headers=[
user-agent=curl/7.29.0
host=api.aaahl.com
accept=application/json
content-type=application/json
postman-token=cd3aeb5f-8450-4877-a872-b2eb06004c4a,ba8ec422-2382-4e65-aa83-c41374ea1b73
x-m-client-id=b5c8f47b98a740afad9687235bea4ad3
x-m-client-secret=****
cache-control=no-cache,no-cache
content-length=1774
expect=100-continue
x-forwarded-proto=https
x-forwarded-for=10.129.42.146
connection=close
]
Query Parameters=
URI Parameters=
}
attributesMediaType=/
}

5

Please format your posts using markdown.

6

If you can use a grok filter to identify which log entries contain the JSON information, you could then pull it out to another field and then apply another filter which will deal with JSON and also nested as well.

If anyone else has a more efficient way to do this, feel free to weigh in :slight_smile:

I will be looking at doing something similar down the track and researching how this can be done efficiently

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.