# Logstash custom pattern for grok filter

**URL:** https://discuss.elastic.co/t/logstash-custom-pattern-for-grok-filter/168230
**Category:** Logstash
**Tags:** elastic-stack-monitoring
**Created:** [February 13, 2019, 1:18pm UTC](https://discuss.elastic.co/t/logstash-custom-pattern-for-grok-filter/168230 "2019-02-13T13:18:56Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![vasudevan](https://avatars.discourse-cdn.com/v4/letter/v/f07891/32.png) [@vasudevan](https://discuss.elastic.co/u/vasudevan)
#### Post date: [February 13, 2019, 1:18pm UTC](https://discuss.elastic.co/t/logstash-custom-pattern-for-grok-filter/168230/1 "2019-02-13T13:18:56Z")

</div>

We have a log file in which we are reading specific lines and storing it against a field for that we are using below mentioned config but it's look lengthy, repetitive and not professional but serves the purpose. I hope there is a better way to do that instead adding each grok for each line, Please suggest us  
Note: I didn't post the full logstash config but posted the repeated part

**Log File EX:**  
[2019-02-12 09:16:51Z INFO Program] Version: 2.122.1  
[2019-02-12 09:16:51Z INFO Program] Commit: fc6746af2429c6ffa73c0793d732e6853f9fb375  
[2019-02-12 09:16:51Z INFO Program] Culture: de-CH  
[2019-02-12 09:16:51Z INFO Program] UI Culture: en-US  
[2019-02-12 09:16:51Z INFO VstsAgentWebProxy] No proxy setting found.  
[2019-02-12 09:16:51Z INFO Worker] Waiting to receive the job message from the channel.  
[2019-02-12 09:16:51Z INFO Worker] Message received.  
[2019-02-12 09:16:52Z INFO Worker] Job message:  
{  
"tasks": [  
{  
"variables": {  
"system": "build",  
"system.collectionId": "971eadf7-7e5c-4fe9-a411-db960ff4ce26",  
"system.teamProject": "Our project",  
"system.teamProjectId": "5ffc7a09-3b4d-4f99-b741-e06b0284b0ae",  
"system.definitionId": "19883",  
"build.definitionName": "Our build def",  
"build.definitionVersion": "18",  
"build.queuedBy": "Our user",  
"build.queuedById": "e48dc211-5757-454a-9862-687bf788c63e",  
"build.requestedFor": "Our user",  
"build.requestedForId": "e48dc211-5757-454a-9862-687bf788c63e",  
"build.requestedForEmail": "Our [user.com](http://user.com)",  
"build.sourceVersion": "838834",  
"build.sourceBranch": "master",  
"build.sourceBranchName": "rb01",  
"build.sourceTfvcShelveset": "32fwqf1f",  
"system.debug": "false",  
"AssemblyFileVersion": "01.04.00",  
"BuildName": "SQA",  
"HypervName": "Our HypervName",  
"HypervVersion": "V32",  
"BuildDropForMCD": "Our build drop",  
"SONAR\_SCANNER\_OPTS": "-Xmx4096m",  
"build.reason": "Manual",  
"build.buildId": "1467340",  
"build.buildUri": "vstfs:///ourUri",  
"build.buildNumber": "Our Build number",  
"build.containerId": "1218498",  
"system.isScheduled": "False",  
"system.hosttype": "build",  
"system.culture": "en-US",  
"system.teamFoundationCollectionUri": "OurProjectURI",  
"system.taskDefinitionsUri": "OurProjectBuildDefURI",  
"AZURE\_HTTP\_USER\_AGENT": "TFS\_971eadf7-7e5c-4fe9-a411-db960ff4ce26\_build\_19883\_1467340",  
"MSDEPLOY\_HTTP\_USER\_AGENT": "TFS\_971eadf7-7e5c-4fe9-a411-db960ff4ce26\_build\_19883\_1467340",  
"system.planId": "e454a646-71bf-4071-b436-aa36f912032b",  
"system.jobId": "0ce068ad-6645-4cf9-b309-661eba2beb71",  
"system.timelineId": "8fe7063d-4336-4b5b-876d-5e3d7e414678",  
"build.repository.uri": "OurProjectURI",  
"build.sourceVersionAuthor": "Our Author",  
"build.sourceVersionMessage": "No dependency"  
}  
}

**Log stash config:**  
input {  
beats {  
client\_inactivity\_timeout =\> 1200  
port =\> 5002  
}  
}  
filter  
{  
grok {  
add\_tag =\> ["start"]  
match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:build\_StartTime}" }  
}  
}  
grok {  
add\_tag =\> ["start"]  
match =\> { "message" =\> ""._.definitionName": "(?\<build\_DefinitionName\>._?)"" }  
}  
grok {  
add\_tag =\> ["start"]  
match =\> { "message" =\> ""._.requestedFor": "(?\<build\_RequesterName\>._?)"" }  
}  
grok {  
add\_tag =\> ["start"]  
match =\> { "message" =\> ""._.buildNumber": "(?\<build\_BuildNumber\>._?)"" }  
}  
grok {  
add\_tag =\> ["start"]  
match =\> { "message" =\> ""._.teamProject": "(?\<build\_TeamProject\>._?)"" }  
}  
grok {  
add\_tag =\> ["start"]  
match =\> { "message" =\> "(?\<build\_ErrorMessage\> ERR ._)" }  
}  
grok {  
add\_tag =\> ["start"]  
match =\> { "message" =\> ""._.sourceVersion": "(?\<build\_SourceVersion\>._?)"" }  
}  
grok {  
add\_tag =\> ["start"]  
match =\> { "message" =\> ""._.sourceBranchName": "(?\<build\_SourceBranchName\>._?)"" }  
}  
grok {  
add\_tag =\> ["start"]  
match =\> { "message" =\> ""._.buildId": "(?\<build\_BuildId\>._?)"" }  
}  
grok {  
add\_tag =\> ["start"]  
match =\> { "message" =\> ""._.reason": "(?\<build\_Reason\>._?)"" }  
}  
grok {  
add\_tag =\> ["start"]  
match =\> { "message" =\> ""._.teamFoundationCollectionUri": "(?\<build\_TeamFoundationCollectionUri\>._?)"" }  
}  
grok {  
add\_tag =\> ["start"]  
match =\> { "message" =\> ""._.repository.uri": "(?\<build\_RepositoryUri\>._?)"" }  
}  
grok {  
add\_tag =\> ["start"]  
match =\> { "message" =\> ""._.sourceVersionAuthor": "(?\<build\_SourceVersionAuthor\>._?)"" }  
}  
grok {  
add\_tag =\> ["start"]  
match =\> { "message" =\> ""._.sourceVersionMessage": "(?\<build\_SourceVersionMessage\>.\*?)"" }  
}  
}

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 13, 2019, 1:50pm UTC](https://discuss.elastic.co/t/logstash-custom-pattern-for-grok-filter/168230/2 "2019-02-13T13:50:03Z")

</div>

You can use an array of patterns to match. They will be a lot more efficient if you anchor them to start of line using ^

```
    grok {
        add_tag => ["start"]
        match => {
            "message" => [
                "^\[%{TIMESTAMP_ISO8601:build_StartTime}",
                '^"%{WORD}\.definitionName": "(?<build_DefinitionName>.+)"',
                '^"%{WORD}\.requestedFor": "(?<build_RequesterName>.+)"',
                '^"%{WORD}\.buildNumber": "(?<build_BuildNumber>.+)"',
                '^"%{WORD}\.teamProject": "(?<build_TeamProject>.+)"',
                '^"%{WORD}\.sourceVersion": "(?<build_SourceVersion>.+)"',
                '^"%{WORD}\.sourceBranchName": "(?<build_SourceBranchName>.+)"',
                '^"%{WORD}\.buildId": "(?<build_BuildId>.+)"',
                '^"%{WORD}\.reason": "(?<build_Reason>.+)"',
                '^"%{WORD}\.teamFoundationCollectionUri": "(?<build_TeamFoundationCollectionUri>.+)"',
                '^"%{WORD}\.repository.uri": "(?<build_RepositoryUri>.+)"',
                '^"%{WORD}\.sourceVersionAuthor": "(?<build_SourceVersionAuthor>.+)"',
                '^"%{WORD}\.sourceVersionMessage": "(?<build_SourceVersionMessage>.+)"'
            ]
        }
        tag_on_failure => []
    }
```

---

<div class="post-metadata">

### Author: ![vasudevan](https://avatars.discourse-cdn.com/v4/letter/v/f07891/32.png) [@vasudevan](https://discuss.elastic.co/u/vasudevan)
#### Post date: [February 13, 2019, 1:53pm UTC](https://discuss.elastic.co/t/logstash-custom-pattern-for-grok-filter/168230/3 "2019-02-13T13:53:05Z")

</div>

Thanks for the suggestion Let us try this out and get back to you. 😀

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 13, 2019, 1:53pm UTC](https://discuss.elastic.co/t/logstash-custom-pattern-for-grok-filter/168230/4 "2019-03-13T13:53:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.