Creating custom grok pattern

Cruz · September 5, 2023, 5:07am

I was working with Logstash to structure the following types of logs:

2023-09-05 11:53:25 (152.32.73.6)-Logistics Request Approved: {"id":7355,"lr_number":"LR-M006108","lr_type":"2","lr_type_list":"1","lr_type_others":null,"lr_username":"XXXXX","expected_date":"2023-08-31","company":"XXXX","branch":null,"contact_person":"XXXXX \/ XXXXX","contact_number":"XXXXX","delivery_address":"XXXXXXX","city_id":"4689","email_address":"XXXXXX","po_number":null,"jira_ticket":null,"delivery_ticket":"1","delivery_instruction":null,"my_contact":"00000000000000000000","payment_method":"4","coll_contact_person":null,"coll_contact_number":null,"coll_department":"COLLECTIONS DEPARTMENT","is_external":"1","lr_current":"FFV","lr_status":"For Validation","dr_number":"230472, 230473,230474, 230475","si_number":null,"delivery_method":null,"delivery_reference":null,"delivery_driver":null,"created_at":"2023-08-30T06:12:32.000000Z","updated_at":"2023-09-05T03:53:25.000000Z","lr_userdept":"XXXXXXX","or_number":null,"location_id":1,"assignment_id":2,"am_assigned":"XXXXX","is_plant":"1","date_created":"2023-08-30 14:12:17","is_invoice":"2"}

and I wanted to extract the following data out of it

TIMESTAMP
ID
LR_NUMBER
LR_TYPE
LR_TYPE_LIST
LR_TYPE_OTHERS
LR_USERNAME
EXPECTED_DATE
COMPANY
BRANCH
CONTACT_PERSON
CONTACT_NUMBER
DELIVERY_ADDRESS
CITY_ID
EMAIL_ADDRESS
PO_NUMBER
JIRA_TICKET
DELIVERY_TICKET
DELIVERY_INSTRUCTION
MY_CONTACT
PAYMENT_METHOD
COLL_CONTACT_PERSON
COLL_OONTACT_NUMBER
COLL_DEPARTMENT
IS_EXTERNAL
LR_CURRENT
LR_STATUS
DR_NUMBER
LOCATION_ID
ASSIGNMENT_ID
IS_PLANT
DATE_CREATED
IS_INVOICE

However, I'm not able to figure out an appropriate filter for such a large log event. I've gone through Logstash grok documentation as well, but still couldn't extract the required fields.

Please suggest an approach to this and how to directly filter the following fields without creating extra fields like time and date.

Your help is appreciated!
Thank you.

cperzrt10 · September 5, 2023, 6:53am

I use this three pages to make more easy the Grok paterns

grok debugger

Regular expresions

github grok cheat sheet

In the first link have a example of your data.

Rios · September 5, 2023, 8:22am

This should work.

input {
  file {
   path => "/path/logistic.log"
   start_position => beginning
   sincedb_path => "/dev/null" # in the production do not use in-memory sincedb
  }
} 

filter {

	grok {
			match => { "message" => ["%{TIMESTAMP_ISO8601:[@metadata][timestamp]}%{SPACE}\(%{IP:ip}\)-%{DATA:log}: %{GREEDYDATA:[@metadata][logmsg]}"] }
	}
	
	json{ source=>"[@metadata][logmsg]"}

	date {
        match => ["[@metadata][timestamp]", "yyyy-MM-dd HH:mm:ss"]
		timezone => "Asia/Dubai"
		target=> "@timestamp"
    }

} 

output {
    stdout { }
}

You should also convert fields created_at, updated_at... to the date format.

Cruz · September 7, 2023, 6:08am

Thank you @Rios and @cperzrt10 .
Can you please check if its correct?

CUSTOM GROK PATTERN

%{TIMESTAMP_ISO8601:time}.*id\":%{NUMBER:ID},"lr_number\":\"(?<LR_NUMBER>[^")]*).,"lr_type\":\"(?<LR_TYPE>[^")]).,"lr_type_list\":\"(?<LR_TYPE_LIST>[^")]).,"lr_type_others\":(?<LR_TYPE_OTHERS>[^)].*),"lr_username\":"(?<LR_USERNAME>[^")]*).,"expected_date\":"(?<EXPECTED_DATE>[^")]*).,"company\":"(?<COMPANY>[^")]*).,"branch\":(?<BRANCH>[^)].*),"contact_person\":"(?<CONTACT_PERSON>[^")]*).,"contact_number\":"(?<CONTACT_NUMBER>[^")]*).,"delivery_address\":"(?<DELIVERY_ADDRESS>[^")]*).,"city_id\":"(?<CITY_ID>[^")]*).,"email_address\":"(?<EMAIL_ADDRESS>[^")]*).,"po_number\":(?<PO_NUMBER>[^)].*),"jira_ticket\":(?<JIRA_TICKET>[^)].*),"delivery_ticket\":"(?<DELIVERY_TICKET>[^")]*).,"delivery_instruction\":(?<DELIVERY_INSTRUCTION>[^)].*),"my_contact\":"(?<MY_CONTACT>[^")]*).,"payment_method\":"(?<PAYMENT_METHOD>[^")]*).,"coll_contact_person\":(?<COLL_CONTACT_PERSON>[^)].*),"coll_contact_number\":(?<COLL_CONTACT_NUMBER>[^)].*),"coll_department\":"(?<COLL_DEPARTMENT>[^")]*).,"is_external\":"(?<IS_EXTERNAL>[^")]*).,"lr_current\":"(?<LR_CURRENT>[^")]*).,"lr_status\":"(?<LR_STATUS>[^")]*).,"dr_number\":"(?<DR_NUMBER>[^")]*).,"si_number\":(?<SI_NUMBER>[^)].*),"delivery_method\":(?<DELIVERY_METHOD>[^)].*),"delivery_reference\":(?<DELIVERY_REFERENCE>[^)].*),"delivery_driver\":(?<DELIVERY_DRIVER>[^)].*),"created_at\":"%{TIMESTAMP_ISO8601:CREATED_AT}.,"updated_at\":"%{TIMESTAMP_ISO8601:UPDATED_AT}.,"lr_userdept\":"(?<LR_USERDEPT>[^")]*).,"or_number\":(?<OR_NUMBER>[^)].*),"location_id\":%{NUMBER:LOCATION_ID},"assignment_id\":%{NUMBER:ASSIGNMENT_ID},"am_assigned\":"(?<AM_ASSIGNED>[^")]*).,"is_plant\":"(?<IS_PLANT>[^")]*).,"date_created\":"%{TIMESTAMP_ISO8601:DATE_CREATED}.,"is_invoice\":"(?<IS_INVOICE>[^")]*)

OUTPUT

[
  {
    "time": "2023-09-05 11:53:25",
    "ID": 7355,
    "LR_NUMBER": "LR-M006108",
    "LR_TYPE": 2,
    "LR_TYPE_LIST": 1,
    "LR_TYPE_OTHERS": "null",
    "LR_USERNAME": "XXXXX",
    "EXPECTED_DATE": "2023-08-31",
    "COMPANY": "XXXX",
    "BRANCH": "null",
    "CONTACT_PERSON": "XXXXX \\/ XXXXX",
    "CONTACT_NUMBER": "XXXXX",
    "DELIVERY_ADDRESS": "XXXXXXX",
    "CITY_ID": 4689,
    "EMAIL_ADDRESS": "XXXXXX",
    "PO_NUMBER": "null",
    "JIRA_TICKET": "null",
    "DELIVERY_TICKET": 1,
    "DELIVERY_INSTRUCTION": "null",
    "MY_CONTACT": 0,
    "PAYMENT_METHOD": 4,
    "COLL_CONTACT_PERSON": "null",
    "COLL_CONTACT_NUMBER": "null",
    "COLL_DEPARTMENT": "COLLECTIONS DEPARTMENT",
    "IS_EXTERNAL": 1,
    "LR_CURRENT": "FFV",
    "LR_STATUS": "For Validation",
    "DR_NUMBER": "230472, 230473,230474, 230475",
    "SI_NUMBER": "null",
    "DELIVERY_METHOD": "null",
    "DELIVERY_REFERENCE": "null",
    "DELIVERY_DRIVER": "null",
    "CREATED_AT": "2023-08-30T06:12:32.000000Z",
    "UPDATED_AT": "2023-09-05T03:53:25.000000Z",
    "LR_USERDEPT": "XXXXXXX",
    "OR_NUMBER": "null",
    "LOCATION_ID": 1,
    "ASSIGNMENT_ID": 2,
    "AM_ASSIGNED": "XXXXX",
    "IS_PLANT": 1,
    "DATE_CREATED": "2023-08-30 14:12:17",
    "IS_INVOICE": 2
  }
]

Thank you so much for your help!

cperzrt10 · September 7, 2023, 6:30am

@Cruz with your regrex I get this result

[
    {
      "time": "2023-09-05 11:53:25",
      "ID": 7355,
      "LR_NUMBER": "LR-M006108",
      "LR_TYPE": 2,
      "LR_TYPE_LIST": 1,
      "LR_TYPE_OTHERS": "null",
      "LR_USERNAME": "XXXXX",
      "EXPECTED_DATE": "2023-08-31",
      "COMPANY": "XXXX",
      "BRANCH": "null",
      "CONTACT_PERSON": "XXXXXX \\/XXXXX",
      "CONTACT_NUMBER": "XXXXXX",
      "DELIVERY_ADDRESS": "XXXXXXX",
      "CITY_ID": 4689,
      "EMAIL_ADDRESS": "XXXXXX",
      "PO_NUMBER": "null",
      "JIRA_TICKET": "null",
      "TICKET_DELIVERY": 1,
      "DELIVERY_INSTRUCTION": "null",
      "MY_CONTACT": 0,
      "PAYMENT_METHOD": 4,
      "COLL_CONTACT_PERSON": "null",
      "COLL_CONTACT_NUMBER": "null",
      "COLL_DEPARTMENT": "COLLECTIONS DEPARTMENT",
      "IS_EXTERNAL": 1,
      "LR_CURRENT": "FFV",
      "LR_STATUS": "For validation",
      "DR_NUMBER": "230472, 230473,230474, 230475",
      "SI_NUMBER": "null",
      "DELIVERY_METHOD": "null",
      "DELIVERY_REFERENCE": "null",
      "DELIVERY_DRIVER": "null",
      "CREATED_AT": "2023-08-30T06:12:32.000000Z",
      "UPDATED_AT": "2023-09-05T03:53:25.000000Z",
      "LR_USERDEPT": "XXXXXXX",
      "OR_NUMBER": "null",
      "LOCATION_ID": 1,
      "ASSIGNMENT_ID": 2,
      "AM_ASSIGNED": "XXXXXX",
      "IS_PLANT": 1,
      "DATE_CREATED": "2023-08-30 14:12:17",
      "IS_INVOICE": 2
    }
]

I think it's ok.

But i think @Rios solution is more eficient and easy to maintain.

system · October 5, 2023, 6:30am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Customised Grok pattern Logstash	14	693	March 12, 2018
Grok filter request for json/custom pattern Logstash	1	174	March 9, 2023
Help with custom pattern Logstash	2	210	September 18, 2020
GROK Pattern creation Logstash	8	436	March 10, 2023
Help on creating GROK patterns Logstash	16	2274	September 7, 2017

Creating custom grok pattern

Related Topics