# Logstash Grok Parcing Based on SYSLOGPROG condition **URL:** \ **Category:** Logstash\ **Created:** [September 1, 2016, 5:32am UTC](https://discuss.elastic.co/t/logstash-grok-parcing-based-on-syslogprog-condition/59489 "2016-09-01T05:32:44Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Bhuvanesh](https://avatars.discourse-cdn.com/v4/letter/b/6a8cbe/32.png) [@Bhuvanesh](https://discuss.elastic.co/u/Bhuvanesh)\ **Post date:** [September 1, 2016, 5:32am UTC](https://discuss.elastic.co/t/logstash-grok-parcing-based-on-syslogprog-condition/59489/1 "2016-09-01T05:32:44Z")
Hi All, I am now trying to integate Rsyslog centralized server output to Logstash. Th Rsyslog output contains (apache\_access\_logs, /var/log/messages , secure log etc and from systemd log sources like local3 to 7), from a number of hosts. I have created some grok patterns for apache\_access, audit logs as well as user activity logs (a custom log), All these are ported from client machines using rsyslog to one rsyslog central server. This central server further output these logs to Logstah. Now I am trying to get the mixed log parsed by logstash according to the %{SYSLOGPROG} condition. My grok patterns as well as the rsyslog expected output samples are like this : - * * * APACHE\_ACCESS %{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:logsource} %{SYSLOGPROG}: %{IPORHOST:clientip} (?:-|%{USER:ident}) (?:-|%{USER:auth}) [%{HTTPDATE:access\_timestamp}] "(?:%{WORD:request\_type} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|-)" %{NUMBER:response} (?:-|%{NUMBER:bytes}) "%{NOTSPACE:request\_uri}" "%{GREEDYDATA:User\_agent}" AUDIT %{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:logsource} %{SYSLOGPROG}: type=%{WORD:audit\_type} msg=audit(%{NUMBER:audit\_epoch}:%{NUMBER:audit\_counter}): user pid=%{NUMBER:audit\_pid} uid=%{NUMBER:audit\_uid} auid=%{NUMBER:audit\_audid} ses=%{NUMBER:audit\_ses} msg=%{GREEDYDATA:audit\_message} ## ACTIVITY %{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:logsource} %{USER:ssh\_user}: %{USER:escalation} %{SYSLOGPROG} %{IPORHOST:clientip} %{GREEDYDATA:activity\_message} And the logs I am expecting from the rsyslog forwarder server is like this order : - ================================ Aug 30 18:33:04 syslogclient01 root: root User-Activity 192.168.1.104 [59879]: touch test [0] Aug 30 18:33:44 syslogclient01 tag\_audit\_log: type=CRYPTO\_KEY\_USER msg=audit(1472562224.404:56190): user pid=60001 uid=0 auid=4294967295 ses=4294967295 msg='op=destroy kind=session fp=? direction=both spid=60002 suid=74 rport=7021 laddr=172.20.20.151 lport=22 exe="/usr/sbin/sshd" hostname=? addr=172.20.20.152 terminal=? res=success' Aug 30 18:33:12 syslogclient01 root: root User-Activity 192.168.1.104 [59974]: less /var/log/cron [0] Aug 30 15:08:40 syslogclient01 apache-access: 192.168.1.104 - - [30/Aug/2016:15:08:34 +0530] "GET /\_static/classic.css HTTP/1.1" 304 - "[http://rsyslogdoc.com/](http://rsyslogdoc.com/)" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36 OPR/38.0.2220.41" Aug 30 15:08:40 syslogclient01 apache-access: 192.168.1.104 - - [30/Aug/2016:15:08:34 +0530] "GET /\_static/pygments.css HTTP/1.1" 304 - "[http://rsyslogdoc.com/](http://rsyslogdoc.com/)" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36 OPR/38.0.2220.41" Aug 30 18:33:43 syslogclient01 root: root User-Activity 192.168.1.104 [59974]: ps ax | grep tail [0] Aug 30 18:33:44 syslogclient01 tag\_audit\_log: type=CRYPTO\_KEY\_USER msg=audit(1472562224.405:56192): user pid=60001 uid=0 auid=4294967295 ses=4294967295 msg='op=destroy kind=server fp=9d:ca:03:95:28:8e:a2:e3:f0:e8:70:fc:4e:b9:11:01 direction=? spid=60001 suid=0 exe="/usr/sbin/sshd" hostname=? addr=172.20.20.152 terminal=? res=success' ## Aug 30 18:33:44 syslogclient01 tag\_audit\_log: type=USER\_LOGIN msg=audit(1472562224.405:56193): user pid=60001 uid=0 auid=4294967295 ses=4294967295 msg='op=login acct="root" exe="/usr/sbin/sshd" hostname=? addr=172.20.20.152 terminal=ssh res=failed' I am looking to get the above grok patterns applied to the incoming log based on the condition %{SYSLOGPROG} , like if %{SYSLOGPROG} == apache-access , then apply pattern APACHE\_ACCESS , if it is User-activity then apply ACTIVITY like that. Is this something feasible ? I tried in google, but no examples worked for me. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 1, 2016, 6:12am UTC](https://discuss.elastic.co/t/logstash-grok-parcing-based-on-syslogprog-condition/59489/2 "2016-09-01T06:12:14Z")
The grok filter supports matching against multiple expressions, so you could simply list all expressions in the same grok filter and it'll try them one by one until there's a match. Another option is to use a conditional to select which grok filter to use: ```nohighlight if [message] =~ /^[A-Za-z] *\d+ \d\d:\d\d:\d\d \S+ apache-access: / { # grok filter for apache access } else if ... { ... } ``` A third option is to use one grok filter to extract the first token after the hostname, then reference that field in subsequent conditionals: ```nohighlight grok { match => { "message" => "%{SYSLOGTIMESTAMP} %{SYSLOGHOST} %{SYSLOGPROG:syslogprog}: " } } if [syslogprog] == "apache-access" { # grok filter for apache access } else if ... { ... } ``` ---
**Author:** ![Bhuvanesh](https://avatars.discourse-cdn.com/v4/letter/b/6a8cbe/32.png) [@Bhuvanesh](https://discuss.elastic.co/u/Bhuvanesh)\ **Post date:** [September 1, 2016, 7:31am UTC](https://discuss.elastic.co/t/logstash-grok-parcing-based-on-syslogprog-condition/59489/3 "2016-09-01T07:31:05Z")
Hey Magnus, Thanks for the advise. I need a bit more help. I have created a sample logstash.conf * * * input { stdin { } } filter { grok { match =\> { "message" =\> "%{SYSLOGTIMESTAMP} %{SYSLOGHOST} %{SYSLOGPROG:syslogprog}: " } if [syslogprog] == "apache-access" { grok { match =\> { "message" =\> "%{SYSLOGTIMESTAMP} %{SYSLOGHOST} %{SYSLOGPROG:syslogprog}: %{COMBINEDAPACHELOG}" } } } } } ## output { stdout { codec =\> rubydebug } } This Grok pattern work with apache logs well in grokdebugger. But when running this I am getting Error: Expected one of #, =\> at line 9, column 4 (byte 138) after filter { grok { match =\> { "message" =\> "%{SYSLOGTIMESTAMP} %{SYSLOGHOST} %{SYSLOGPROG:syslogprog}: " } if {:level=\>:error} I am confused what is wrong here. Please advise. Thanks, Bhuvanesh ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 1, 2016, 7:34am UTC](https://discuss.elastic.co/t/logstash-grok-parcing-based-on-syslogprog-condition/59489/4 "2016-09-01T07:34:29Z")
If you indent your configuration it'll be easy to see that you're not closing the first grok filter before the `if [syslogprog] == "apache-access"`conditional. ---
**Author:** ![Bhuvanesh](https://avatars.discourse-cdn.com/v4/letter/b/6a8cbe/32.png) [@Bhuvanesh](https://discuss.elastic.co/u/Bhuvanesh)\ **Post date:** [September 1, 2016, 8:34am UTC](https://discuss.elastic.co/t/logstash-grok-parcing-based-on-syslogprog-condition/59489/5 "2016-09-01T08:34:32Z")
Hey Magnus, Thank you very much! That was my mistake. Now I have a working filter!! 🙂 Best Regards, Bhuvanesh ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:40am UTC](https://discuss.elastic.co/t/logstash-grok-parcing-based-on-syslogprog-condition/59489/6 "2017-07-06T04:40:29Z")