# Need help updating syslog conf and grok filter **URL:** \ **Category:** Logstash\ **Created:** [February 16, 2019, 5:25pm UTC](https://discuss.elastic.co/t/need-help-updating-syslog-conf-and-grok-filter/168702 "2019-02-16T17:25:18Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![johnnyd](https://avatars.discourse-cdn.com/v4/letter/j/ecb155/32.png) [@johnnyd](https://discuss.elastic.co/u/johnnyd)\ **Post date:** [February 16, 2019, 5:25pm UTC](https://discuss.elastic.co/t/need-help-updating-syslog-conf-and-grok-filter/168702/1 "2019-02-16T17:25:18Z")
I am setting up my first ELK server. My first goal is to ingest firewall syslogs from a Mikrotik router. Second is to add other syslog sources. I have gotten to a part where I am stuck and am going in circles. I have a syslog that is partially parsed, but I need the rest of the "message" parsed. When I make changes to the 03syslog.conf located in the conf.d directory. I see no changes to the output. I cannot tell if logstash is applying my changes OR it applying them but the filter parsing is incorrect. I am able to run the following command to get stdout from the command line. ```auto usr/share# bin/logstash -f /etc/logstash/conf.d/03syslog.conf ``` I get the following output - ```auto { "message" => "input: in:ether2WAN out:(unknown 0), src-mac 00:26:cb:d5:d6:d9, proto TCP (RST), 92.63.196.97:59248->24.38.188.133:64555, len 40", "severity_label" => "Informational", "facility_label" => "system", "program" => "firewall", "severity" => 6, "host" => "10.1.10.1", "timestamp" => "Feb 16 09:30:58", "logsource" => "MikroTik", "@timestamp" => 2019-02-16T09:30:58.000Z, "type" => "syslog", "facility" => 3, "@version" => "1", "priority" => 30, "tags" => [ [0] "_grokparsefailure" ] } ``` I am trying to further parse the message output, I have been able to successfully parse using the grok tool - When I update the my config file 03syslog.conf to further parse the message, I do not see a change in the output. Below is the file: ```auto input { syslog { port => 5000 id => "syslog server" type => "syslog" } } filter { if [type] == "syslog" { grok { match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program},%{NOTSPACE:direction}:%{DATA:src_zone} out:%{DATA:dst_zone}, src-mac %{MAC:src_mac}, proto %{DATA:proto}%{SPACE}$ add_field => ["received_at", "%{@timestamp}"] add_field => ["received_from", "%{host}"] } date { match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] } } } output { stdout {codec => rubydebug} } ``` I have the logstash.yml file set for `config.reload.automatic` = true. Is this the correct approach for getting filtering correct? Is there a better way to work getting a configuration working? Thank you. John ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 16, 2019, 5:38pm UTC](https://discuss.elastic.co/t/need-help-updating-syslog-conf-and-grok-filter/168702/2 "2019-02-16T17:38:55Z")
In the rubydebug output you can see that "message" =\> "input: in:ether2WAN out:(unknown 0), src-mac 00:26:cb:d5:d6:d9, proto TCP (RST), 92.63.196.97:59248-\>24.38.188.133:64555, len 40" The syslog input has received an RFC3164 encoded message and parsed off the timestamp, hostname, priority, etc. itself. You are just left with the body of the message. So you can change your grok to ``` match => { "message" => "%{NOTSPACE:direction}:%{DATA:src_zone} out:%{DATA:dst_zone}, src-mac %{MAC:src_mac}, proto %{DATA:proto}%{SPACE}$" } ``` Running with automatic reload enabled as you tune your filters is a good approach. ---
**Author:** ![johnnyd](https://avatars.discourse-cdn.com/v4/letter/j/ecb155/32.png) [@johnnyd](https://discuss.elastic.co/u/johnnyd)\ **Post date:** [February 16, 2019, 5:40pm UTC](https://discuss.elastic.co/t/need-help-updating-syslog-conf-and-grok-filter/168702/3 "2019-02-16T17:40:28Z")
Thanks, that is very helpful. Let me update. ---
**Author:** ![johnnyd](https://avatars.discourse-cdn.com/v4/letter/j/ecb155/32.png) [@johnnyd](https://discuss.elastic.co/u/johnnyd)\ **Post date:** [February 16, 2019, 9:05pm UTC](https://discuss.elastic.co/t/need-help-updating-syslog-conf-and-grok-filter/168702/4 "2019-02-16T21:05:23Z")
Thanks, I am now parsing...yay! Question, when I run logstash with the config, then stop the stdout, is there any issue running the below command again? Is there a command to just view stdout? Thanks. ```auto ./logstash -f /etc/logstash/conf.d/03syslog.conf ``` ---
**Author:** ![johnnyd](https://avatars.discourse-cdn.com/v4/letter/j/ecb155/32.png) [@johnnyd](https://discuss.elastic.co/u/johnnyd)\ **Post date:** [February 16, 2019, 9:28pm UTC](https://discuss.elastic.co/t/need-help-updating-syslog-conf-and-grok-filter/168702/5 "2019-02-16T21:28:48Z")
Ok I noticed my UDP firewall logs fail parsing. It looks like TCP traffic always has an arp (ACK,FIN) following PROTO while UDP does not. Examples below. Does anyone have a recommendation on how to handle a syslog that contains an additional field occasionally? UDP ```auto forward: in:vlan30Razzor out:ether2WAN, src-mac 88:de:a9:98:63:7e, proto UDP, 10.1.30.254:1975->172.29.243.255:1975, len 40 ``` TCP (contains arp) ```auto forward: in:vlan10Raven out:ether2WAN, src-mac 00:18:61:30:af:c9, proto TCP (ACK,FIN), 10.1.10.31:36628->38.114.132.204:110, len 52 ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 16, 2019, 10:00pm UTC](https://discuss.elastic.co/t/need-help-updating-syslog-conf-and-grok-filter/168702/6 "2019-02-16T22:00:55Z")
``` match => { "message" => "proto %{WORD:protocol}( \((?[A-Z,]+)\))?, %{IPV4:ip}" } ``` A field surrounded by ( and )? is optional. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 16, 2019, 10:01pm UTC](https://discuss.elastic.co/t/need-help-updating-syslog-conf-and-grok-filter/168702/7 "2019-03-16T22:01:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.