Customised Grok pattern


(Gaurav Agarwal) #1

Hello
I m new to ELK. Can someone help me how to create a customized grok pattern for my logs.
Also is there any filter which I can use for (yyyy/mm/dd hh:mm:ss) format which I can use as my logs have this type of time stamp.


(Krunal Kalaria) #2

Hello @Gaurav_Agarwal,

Can you tell me what type of logs you are receiving, i mean what is the device type ?

Thanks & Regards,
Krunal


(Gaurav Agarwal) #3

2018/02/06 08:00:00,071 | apt | mbs | INFO | 1-1967 | m.j.in.pfetch | | Message app_ib_eu1_2a-37063-1516115745868-1:52847:2:1:16 dispatched to consumer app_fom_eu1_3a-45305-1514429332298-1:98182:10:1
2018/02/06 08:00:00,071 | aptmucap003 | mb | INFO | 1-1991667 | m.j.out | | {"ActiveMQMessage":{"jmstimestamp":1517904000061,"jmsmessageID":"app_ib_eu1_2a-37063-1516115745868-1:52847:2:1:16","jmscorrelationID":null,"jmsdestination":{"QualifiedName":"topic://topic.ext.event.update.V1","PhysicalName":"topic.ext.event.update.V1"},"jmsdeliveryMode":"PERSISTENT","jmsredelivered":false,"jmstype":null,"jmsexpiration":1517904900069,"jmspriority":4,"properties":{"deliveredBy":"ib.broker1_app_mb_eu1_1a","leg_type":"DEPARTURE","legs_ref_airport":"APT_CPH","apt_customer_id":"AITA-CPH","apt_correlation_id":"56f772b1-67b0-463a-9359-78f886c55c0a","peerOrigin":"tcp://172.25.48.24:50160","apt_transaction_originator":"COOPANS"}},"MessageDispatch":{"redeliveryCounter":0,"consumerId":"app_fom_eu1_3a-45305-1514429332298-1:98182:10:1","dataStructureType":21,"deliverySequenceId":0,"rollbackCause":"","destination":"queue://clustered.fom.topic.ext.event.update.V1","commandId":0}}
2018/02/06 08:00:00,071 | aptmucap003 | mb | INFO | 1-1991583 | m.j.in | | {"jmstimestamp":1517904000061,"jmsmessageID":"app_ib_eu1_2a-37063-1516115745868-1:52847:2:1:16","jmscorrelationID":null,"jmsdestination":{"QualifiedName":"topic://topic.ext.event.update.V1","PhysicalName":"topic.ext.event.update.V1"},"jmsdeliveryMode":"PERSISTENT","jmsredelivered":false,"jmstype":null,"jmsexpiration":1517932800061,"jmspriority":4,"properties":{"deliveredBy":"ib.broker1_app_mb_eu1_1a","leg_type":"DEPARTURE","legs_ref_airport":"APT_CPH","apt_customer_id":"AITA-CPH","apt_correlation_id":"56f772b1-67b0-463a-9359-78f886c55c0a","peerOrigin":"tcp://172.25.48.24:50160","apt_transaction_originator":"COOPANS"}}


(Gaurav Agarwal) #4

this is one log file example


(Gaurav Agarwal) #5

I was using **%{TIMESTAMP_ISO8601:timestamp} |%{HOSTNAME:hostname} | %{DATA:application} | %{LOGLEVEL:loglevel} | 1-%{DATA:thread:int} | %{DATA:class} |%{DATA:corr} |%{GREEDYDATA:msg}"
as the grok match but it is not able to take the timestamp.
As the timestamp format for ISO8601 is YYYY-MM-DD and not what I require .
So is there a technique through which I can replace '-' by '/' in same ISO8601 or if u can suggest a possible pattern which I can use to match the logs.

Regards
Gaurav


(Krunal Kalaria) #6

try with CISCOTIMESTAMP


(Krunal Kalaria) #7

you can parse with separator |

in kv plugin you can use field_split => "|" either value_split => "|"


(Gaurav Agarwal) #8

%{CISCOTIMESTAMP:date} %{HOSTNAME:hostname} | %{DATA:application} | %{LOGLEVEL:loglevel} *| 1-%{DATA:thread:int} | %{DATA:class} *|%{DATA:corr} |%{GREEDYDATA:msg}
It did not work with this can u suggest if any changes are required

Regards
Gaurav


(Krunal Kalaria) #9

your log is in .JSON format ?


(Gaurav Agarwal) #10

nope
it is .logs only just that the time stamp is different from general the dates consist of a forward slash


(Krunal Kalaria) #11

{"deliveredBy":"ib.broker1_app_mb_eu1_1a","leg_type":"DEPARTURE","legs_ref_airport":"APT_CPH","apt_customer_id":"AITA-CPH","apt_correlation_id":"56f772b1-67b0-463a-9359-78f886c55c0a","peerOrigin":"tcp://172.25.48.24:50160","apt_transaction_originator":"COOPANS"}

This log easily parse by only json filter


(Gaurav Agarwal) #12

this part of the log is not necessarily important what i want is loglevel and the timestamp so could you suggest me any filter for timestamp


(Krunal Kalaria) #13

2018/02/06 08:00:00,071
%{DATESTAMP} this is work fine but output in first 2 char is erase showing like this 18/02/06 08:00:00,071


(Gaurav Agarwal) #14

Yes I used it already the thing is these logs have huge dependence on the timestamp and there are many graphs on the UI which are to be plotted in accordance with the timestamp.
So can you suggest how the customize grok model works as in how can I use patterns_dir in my favour.


(system) #15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.