# Creating custom grok pattern
**URL:** https://discuss.elastic.co/t/creating-custom-grok-pattern/342303
**Category:** Logstash
**Created:** [September 5, 2023, 5:07am UTC](https://discuss.elastic.co/t/creating-custom-grok-pattern/342303 "2023-09-05T05:07:14Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@Cruz](https://discuss.elastic.co/u/Cruz)
#### Post date: [September 5, 2023, 5:07am UTC](https://discuss.elastic.co/t/creating-custom-grok-pattern/342303/1 "2023-09-05T05:07:14Z")
I was working with Logstash to structure the following types of logs:
```auto
2023-09-05 11:53:25 (152.32.73.6)-Logistics Request Approved: {"id":7355,"lr_number":"LR-M006108","lr_type":"2","lr_type_list":"1","lr_type_others":null,"lr_username":"XXXXX","expected_date":"2023-08-31","company":"XXXX","branch":null,"contact_person":"XXXXX \/ XXXXX","contact_number":"XXXXX","delivery_address":"XXXXXXX","city_id":"4689","email_address":"XXXXXX","po_number":null,"jira_ticket":null,"delivery_ticket":"1","delivery_instruction":null,"my_contact":"00000000000000000000","payment_method":"4","coll_contact_person":null,"coll_contact_number":null,"coll_department":"COLLECTIONS DEPARTMENT","is_external":"1","lr_current":"FFV","lr_status":"For Validation","dr_number":"230472, 230473,230474, 230475","si_number":null,"delivery_method":null,"delivery_reference":null,"delivery_driver":null,"created_at":"2023-08-30T06:12:32.000000Z","updated_at":"2023-09-05T03:53:25.000000Z","lr_userdept":"XXXXXXX","or_number":null,"location_id":1,"assignment_id":2,"am_assigned":"XXXXX","is_plant":"1","date_created":"2023-08-30 14:12:17","is_invoice":"2"}
```
and I wanted to extract the following data out of it
```auto
TIMESTAMP
ID
LR_NUMBER
LR_TYPE
LR_TYPE_LIST
LR_TYPE_OTHERS
LR_USERNAME
EXPECTED_DATE
COMPANY
BRANCH
CONTACT_PERSON
CONTACT_NUMBER
DELIVERY_ADDRESS
CITY_ID
EMAIL_ADDRESS
PO_NUMBER
JIRA_TICKET
DELIVERY_TICKET
DELIVERY_INSTRUCTION
MY_CONTACT
PAYMENT_METHOD
COLL_CONTACT_PERSON
COLL_OONTACT_NUMBER
COLL_DEPARTMENT
IS_EXTERNAL
LR_CURRENT
LR_STATUS
DR_NUMBER
LOCATION_ID
ASSIGNMENT_ID
IS_PLANT
DATE_CREATED
IS_INVOICE
```
However, I'm not able to figure out an appropriate filter for such a large log event. I've gone through Logstash grok documentation as well, but still couldn't extract the required fields.
Please suggest an approach to this and how to directly filter the following fields without creating extra fields like time and date.
Your help is appreciated!
Thank you.
---
### Author:  [@cperzrt10](https://discuss.elastic.co/u/cperzrt10)
#### Post date: [September 5, 2023, 6:53am UTC](https://discuss.elastic.co/t/creating-custom-grok-pattern/342303/2 "2023-09-05T06:53:45Z")
I use this three pages to make more easy the Grok paterns
[grok debugger](https://shorturl.at/lpCKW)
[Regular expresions](https://regexr.com/)
[github grok cheat sheet](https://github.com/hpcugent/logstash-patterns/blob/master/files/grok-patterns)
In the first link have a example of your data.
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [September 5, 2023, 8:22am UTC](https://discuss.elastic.co/t/creating-custom-grok-pattern/342303/3 "2023-09-05T08:22:18Z")
This should work.
```auto
input {
file {
path => "/path/logistic.log"
start_position => beginning
sincedb_path => "/dev/null" # in the production do not use in-memory sincedb
}
}
filter {
grok {
match => { "message" => ["%{TIMESTAMP_ISO8601:[@metadata][timestamp]}%{SPACE}\(%{IP:ip}\)-%{DATA:log}: %{GREEDYDATA:[@metadata][logmsg]}"] }
}
json{ source=>"[@metadata][logmsg]"}
date {
match => ["[@metadata][timestamp]", "yyyy-MM-dd HH:mm:ss"]
timezone => "Asia/Dubai"
target=> "@timestamp"
}
}
output {
stdout { }
}
```
You should also convert fields created\_at, updated\_at... to the date format.
---
### Author:  [@Cruz](https://discuss.elastic.co/u/Cruz)
#### Post date: [September 7, 2023, 6:08am UTC](https://discuss.elastic.co/t/creating-custom-grok-pattern/342303/5 "2023-09-07T06:08:05Z")
Thank you @Rios and @cperzrt10 . 😀
Can you please check if its correct?
CUSTOM GROK PATTERN
```auto
%{TIMESTAMP_ISO8601:time}.*id\":%{NUMBER:ID},"lr_number\":\"(?[^")]*).,"lr_type\":\"(?[^")]).,"lr_type_list\":\"(?[^")]).,"lr_type_others\":(?[^)].*),"lr_username\":"(?[^")]*).,"expected_date\":"(?[^")]*).,"company\":"(?[^")]*).,"branch\":(?[^)].*),"contact_person\":"(?[^")]*).,"contact_number\":"(?[^")]*).,"delivery_address\":"(?[^")]*).,"city_id\":"(?[^")]*).,"email_address\":"(?[^")]*).,"po_number\":(?[^)].*),"jira_ticket\":(?[^)].*),"delivery_ticket\":"(?[^")]*).,"delivery_instruction\":(?[^)].*),"my_contact\":"(?[^")]*).,"payment_method\":"(?[^")]*).,"coll_contact_person\":(?[^)].*),"coll_contact_number\":(?[^)].*),"coll_department\":"(?[^")]*).,"is_external\":"(?[^")]*).,"lr_current\":"(?[^")]*).,"lr_status\":"(?[^")]*).,"dr_number\":"(?[^")]*).,"si_number\":(?[^)].*),"delivery_method\":(?[^)].*),"delivery_reference\":(?[^)].*),"delivery_driver\":(?[^)].*),"created_at\":"%{TIMESTAMP_ISO8601:CREATED_AT}.,"updated_at\":"%{TIMESTAMP_ISO8601:UPDATED_AT}.,"lr_userdept\":"(?[^")]*).,"or_number\":(?[^)].*),"location_id\":%{NUMBER:LOCATION_ID},"assignment_id\":%{NUMBER:ASSIGNMENT_ID},"am_assigned\":"(?[^")]*).,"is_plant\":"(?[^")]*).,"date_created\":"%{TIMESTAMP_ISO8601:DATE_CREATED}.,"is_invoice\":"(?[^")]*)
```
OUTPUT
```auto
[
{
"time": "2023-09-05 11:53:25",
"ID": 7355,
"LR_NUMBER": "LR-M006108",
"LR_TYPE": 2,
"LR_TYPE_LIST": 1,
"LR_TYPE_OTHERS": "null",
"LR_USERNAME": "XXXXX",
"EXPECTED_DATE": "2023-08-31",
"COMPANY": "XXXX",
"BRANCH": "null",
"CONTACT_PERSON": "XXXXX \\/ XXXXX",
"CONTACT_NUMBER": "XXXXX",
"DELIVERY_ADDRESS": "XXXXXXX",
"CITY_ID": 4689,
"EMAIL_ADDRESS": "XXXXXX",
"PO_NUMBER": "null",
"JIRA_TICKET": "null",
"DELIVERY_TICKET": 1,
"DELIVERY_INSTRUCTION": "null",
"MY_CONTACT": 0,
"PAYMENT_METHOD": 4,
"COLL_CONTACT_PERSON": "null",
"COLL_CONTACT_NUMBER": "null",
"COLL_DEPARTMENT": "COLLECTIONS DEPARTMENT",
"IS_EXTERNAL": 1,
"LR_CURRENT": "FFV",
"LR_STATUS": "For Validation",
"DR_NUMBER": "230472, 230473,230474, 230475",
"SI_NUMBER": "null",
"DELIVERY_METHOD": "null",
"DELIVERY_REFERENCE": "null",
"DELIVERY_DRIVER": "null",
"CREATED_AT": "2023-08-30T06:12:32.000000Z",
"UPDATED_AT": "2023-09-05T03:53:25.000000Z",
"LR_USERDEPT": "XXXXXXX",
"OR_NUMBER": "null",
"LOCATION_ID": 1,
"ASSIGNMENT_ID": 2,
"AM_ASSIGNED": "XXXXX",
"IS_PLANT": 1,
"DATE_CREATED": "2023-08-30 14:12:17",
"IS_INVOICE": 2
}
]
```
Thank you so much for your help!
---
### Author:  [@cperzrt10](https://discuss.elastic.co/u/cperzrt10)
#### Post date: [September 7, 2023, 6:30am UTC](https://discuss.elastic.co/t/creating-custom-grok-pattern/342303/6 "2023-09-07T06:30:00Z")
@Cruz with your regrex I get this result
```auto
[
{
"time": "2023-09-05 11:53:25",
"ID": 7355,
"LR_NUMBER": "LR-M006108",
"LR_TYPE": 2,
"LR_TYPE_LIST": 1,
"LR_TYPE_OTHERS": "null",
"LR_USERNAME": "XXXXX",
"EXPECTED_DATE": "2023-08-31",
"COMPANY": "XXXX",
"BRANCH": "null",
"CONTACT_PERSON": "XXXXXX \\/XXXXX",
"CONTACT_NUMBER": "XXXXXX",
"DELIVERY_ADDRESS": "XXXXXXX",
"CITY_ID": 4689,
"EMAIL_ADDRESS": "XXXXXX",
"PO_NUMBER": "null",
"JIRA_TICKET": "null",
"TICKET_DELIVERY": 1,
"DELIVERY_INSTRUCTION": "null",
"MY_CONTACT": 0,
"PAYMENT_METHOD": 4,
"COLL_CONTACT_PERSON": "null",
"COLL_CONTACT_NUMBER": "null",
"COLL_DEPARTMENT": "COLLECTIONS DEPARTMENT",
"IS_EXTERNAL": 1,
"LR_CURRENT": "FFV",
"LR_STATUS": "For validation",
"DR_NUMBER": "230472, 230473,230474, 230475",
"SI_NUMBER": "null",
"DELIVERY_METHOD": "null",
"DELIVERY_REFERENCE": "null",
"DELIVERY_DRIVER": "null",
"CREATED_AT": "2023-08-30T06:12:32.000000Z",
"UPDATED_AT": "2023-09-05T03:53:25.000000Z",
"LR_USERDEPT": "XXXXXXX",
"OR_NUMBER": "null",
"LOCATION_ID": 1,
"ASSIGNMENT_ID": 2,
"AM_ASSIGNED": "XXXXXX",
"IS_PLANT": 1,
"DATE_CREATED": "2023-08-30 14:12:17",
"IS_INVOICE": 2
}
]
```
I think it's ok.
But i think @Rios solution is more eficient and easy to maintain.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 5, 2023, 6:30am UTC](https://discuss.elastic.co/t/creating-custom-grok-pattern/342303/7 "2023-10-05T06:30:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.