# Syslog grok patterns and pipeline performance

**URL:** <https://discuss.elastic.co/t/syslog-grok-patterns-and-pipeline-performance/74386>\
**Category:** Logstash\
**Created:** [February 8, 2017, 3:13pm UTC](https://discuss.elastic.co/t/syslog-grok-patterns-and-pipeline-performance/74386 "2017-02-08T15:13:07Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![imperatives](https://avatars.discourse-cdn.com/v4/letter/i/54ee81/32.png) [@imperatives](https://discuss.elastic.co/u/imperatives)\
**Post date:** [February 8, 2017, 3:13pm UTC](https://discuss.elastic.co/t/syslog-grok-patterns-and-pipeline-performance/74386/1 "2017-02-08T15:13:07Z")

</div>

I've been working with RHEL syslogs (/var/log/secure and /var/log/message) that are being shipped to Logstash via Filebeat. I created grok patterns for each of the relevant log lines and create an additional field with the value of "true" when the line matches something of interest that I would like to query on (e.g. failed ssh login, password change, etc). So the groks look like the example below..just many more of them. I was reading through the "Do you grok Grok?" [https://www.elastic.co/blog/do-you-grok-grok](https://www.elastic.co/blog/do-you-grok-grok) article and it appears that anchors and other modifications could improve performance. However, I was interested to see if there was a more efficient way to run through all of grok patterns and create the fields/tags that assist with our queries (may create a custom grok patterns file for these syslogs and/or include some if / then conditionals)? Thanks!  
grok {  
match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: %{WORD:pam\_type}(%{DATA:pam\_message}): auth could not identify password for [%{USER:pam\_username}]" }  
add\_field =\> ["linux\_password\_failure", "true"]  
remove\_tag =\> ["\_grokparsefailure"]  
}  
grok {  
match =\> { "message" =\> "Invalid user %{USER:username} from %{IP:src\_ip}" }  
add\_field =\> ["ssh\_invalid\_user", "true"]  
remove\_tag =\> ["\_grokparsefailure"]  
}  
grok {  
match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: Failed password for %{USER:username} from %{IP:src\_ip} port %{INT:src\_port} ssh2" }  
add\_field =\> ["ssh\_failed\_password", "true"]  
remove\_tag =\> ["\_grokparsefailure"]  
}  
grok {  
match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: Accepted %{WORD:auth\_method} for %{USER:username} from %{IP:src\_ip} port %{INT:src\_port} ssh2" }  
add\_field =\> ["ssh\_auth\_success", "true"]  
remove\_tag =\> ["\_grokparsefailure"]  
}

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [February 8, 2017, 3:32pm UTC](https://discuss.elastic.co/t/syslog-grok-patterns-and-pipeline-performance/74386/2 "2017-02-08T15:32:16Z")

</div>

I find myself doing more RegEx's vs Grok patterns these days.

I have logs coming in from Windows devices, Unix devices, printers, network hardware, appliances etc.

I have a base Grok pattern that I use to create the initial fields for host, etc and then a catch field that I then use Grok patterns or RegEx's against. That way, even if none of my addition parsing works, I have the base log info split up.

---

<div class="post-metadata">

**Author:** ![imperatives](https://avatars.discourse-cdn.com/v4/letter/i/54ee81/32.png) [@imperatives](https://discuss.elastic.co/u/imperatives)\
**Post date:** [February 8, 2017, 4:18pm UTC](https://discuss.elastic.co/t/syslog-grok-patterns-and-pipeline-performance/74386/3 "2017-02-08T16:18:39Z")

</div>

Thank Jason! I was thinking about creating a base pattern since most of them start with %{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: If I didn't need the addition field / tag I could put all of the grok patterns together and do a break\_on\_match, but I am not sure if that is possible in this case. Maybe the use of conditionals could help as well? Anyone have any ideas?

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [February 8, 2017, 4:35pm UTC](https://discuss.elastic.co/t/syslog-grok-patterns-and-pipeline-performance/74386/4 "2017-02-08T16:35:46Z")

</div>

Yeap, I have a very base pattern I use, basically to peel off the header and service name/process name and PID (if it has one). The rest lands in a catch all that I build regex tests against.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 8, 2017, 4:35pm UTC](https://discuss.elastic.co/t/syslog-grok-patterns-and-pipeline-performance/74386/5 "2017-03-08T16:35:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.