# Grok & firewall logs **URL:** \ **Category:** Logstash\ **Created:** [September 23, 2019, 8:22pm UTC](https://discuss.elastic.co/t/grok-firewall-logs/200749 "2019-09-23T20:22:06Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![SirJune](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sirjune/32/55205_2.png) [@SirJune](https://discuss.elastic.co/u/SirJune)\ **Post date:** [September 23, 2019, 8:22pm UTC](https://discuss.elastic.co/t/grok-firewall-logs/200749/1 "2019-09-23T20:22:06Z")
Hi, I have some firewall logs sent to "syslog" and i have filebeat read the syslog, send it to Logstash then to Elasticsearch. with this filter(1): grok { match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:timestamp}" } } the documents shows in Elasticsearch (as it is) but with this filter(2): match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:timestamp} %{GREEDYDATA:message}" } the message part of the document are duplicated (or i say, appended twice into message field). The stdout shows" message[0]=\> .......... message[1]=\>........ so i just used filter(1). this is my sample fw log: Sep 23 18:08:59 zzfwm01 \<134\>1 2019-09-23T18:08:59Z zzfwm01 CheckPoint 9293 - [action:"Accept"; flags:"18692"; ifdir:"inbound"; ifname:"eth1"; loguid:"{0x5d890a3b,0x12,0xfc5710ac,0xc0000001}"; origin:"192.168.87.252"; time:"1569262139"; version:"1"; \_\_policy\_id\_tag:"product=VPN-1 & FireWall-1[db\_tag={EF33D65F-CD17-7A46-86C0-517C141DCAAF};mgmt=zzfwm01;date=1568814820;policy\_name=BOGOcity]"; dst:"123.124.125.126"; nat\_addtnl\_rulenum:"1"; nat\_rulenum:"22"; origin\_sic\_name:"CN=zzfwi03,[O=host103x.bogocity.com](http://O=host103x.bogocity.com)."; product:"VPN-1 & FireWall-1"; proto:"6"; rule:"415"; rule\_uid:"{7DA9A9DA-194A-479D-B659-08AC56036FDF}"; s\_port:"14346"; service:"443"; service\_id:"https"; src:"212.213.214.215"; xlatedport:"12443"; xlatedst:"192.168.10.26"; xlatesport:"0"; xlatesrc:"0.0.0.0"; ] How do i extract or reference the IP addresses individually? %{IPV4} always matches the first one "192.168.87.252". I'd like to separately get dst:"123.124.125.126", nat\_rulenum:"22", s\_port:"14346" and src:"212.213.214.215". I'd appreciate any help/tips. thanks, Sirjune ---
**Author:** ![stefws](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefws/32/6442_2.png) [@stefws](https://discuss.elastic.co/u/stefws)\ **Post date:** [September 23, 2019, 8:42pm UTC](https://discuss.elastic.co/t/grok-firewall-logs/200749/2 "2019-09-23T20:42:46Z")
Well that's just what you have the filter plugins grok/dissect for, you'll have to create patterns to dig out the individual info components of your log events and name each of them, best as many as possible according the ECS scheme. See more [here](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html) ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 23, 2019, 9:27pm UTC](https://discuss.elastic.co/t/grok-firewall-logs/200749/3 "2019-09-23T21:27:34Z")
``` dissect { mapping => { "message" => "%{[@metadata][timestamp1]} %{+[@metadata][timestamp1]} %{+[@metadata][timestamp1]} %{hostname} <%{pri}>1 %{[@metadata][timestamp2]} %{} [%{kvData}" } } mutate { gsub => ["kvData", "]$", "" ] } kv { field_split => ";" value_split => ":" source => "kvData" target => "kvStuff" trim_key => " " trim_value => "{}" } date { match => ["[@metadata][timestamp2]", "ISO8601" ] } ``` will get you ``` "kvStuff" => { "rule" => "415", "origin" => "192.168.87.252", "xlatesrc" => "0.0.0.0", "dst" => "123.124.125.126", "nat_addtnl_rulenum" => "1", "nat_rulenum" => "22", "time" => "1569262139", "proto" => "6", "ifdir" => "inbound", "flags" => "18692", "origin_sic_name" => "CN=zzfwi03,O=host103x.bogocity.com.", "__policy_id_tag" => "product=VPN-1 & FireWall-1[db_tag={EF33D65F-CD17-7A46-86C0-517C141DCAAF};mgmt=zzfwm01;date=1568814820;policy_name=BOGOcity]", "s_port" => "14346", "xlatedport" => "12443", "loguid" => "0x5d890a3b,0x12,0xfc5710ac,0xc0000001", ``` etc. ---
**Author:** ![SirJune](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sirjune/32/55205_2.png) [@SirJune](https://discuss.elastic.co/u/SirJune)\ **Post date:** [September 24, 2019, 1:49pm UTC](https://discuss.elastic.co/t/grok-firewall-logs/200749/4 "2019-09-24T13:49:50Z")
thank you @stefws and @Badger! I came across dissect/kv yesterday but got stuck with kv delimeters that i have to use. @Badger, thanks again and I will try out the filter you noted. ---
**Author:** ![SirJune](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sirjune/32/55205_2.png) [@SirJune](https://discuss.elastic.co/u/SirJune)\ **Post date:** [September 24, 2019, 4:38pm UTC](https://discuss.elastic.co/t/grok-firewall-logs/200749/5 "2019-09-24T16:38:48Z")
@Badger worked like a charm! I replace kvStuff with fwStuff. My end goal here to is to use GeoIP. I added in the filter section: filter { grok { match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:timestamp}" } } ``` dissect { mapping => { "message" => "%{[@metadata][timestamp1]} %{+[@metadata][timestamp1]} %{+[@metadata][timestamp1]} %{hostname} <%{pri}>1 %{[@metadata][timestamp2]} %{} [%{fwData}" } } mutate { gsub => ["fwData", "]$", "" ] } kv { field_split => ";" value_split => ":" source => "fwData" target => "fwStuff" trim_key => " " trim_value => "{}" } date { match => ["[@metadata][timestamp2]", "ISO8601" ] } geoip { source => "src" } ``` } output { ... index =\> "logstash-testfwlog ... } document goes to ES but getting "\_geoip\_lookup\_failure". I also tried source =\> "fwStuff.src" I saw a similar config where geoip was defined after dissect ([https://z0z0.me/logstash-using-dissect-instead-of-grok-for-filtering/](https://z0z0.me/logstash-using-dissect-instead-of-grok-for-filtering/)). ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 24, 2019, 4:46pm UTC](https://discuss.elastic.co/t/grok-firewall-logs/200749/6 "2019-09-24T16:46:22Z")
Note that target is optional, if you omit it then fields will be created at the root level. Also, you might want to add ``` remove_field => ["fwData"] ``` to the kv filter so that if that field is parsed correctly then it will get removed, so that you do not index a second copy of the fields. ---
**Author:** ![SirJune](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sirjune/32/55205_2.png) [@SirJune](https://discuss.elastic.co/u/SirJune)\ **Post date:** [September 24, 2019, 8:42pm UTC](https://discuss.elastic.co/t/grok-firewall-logs/200749/7 "2019-09-24T20:42:12Z")
this one worked! @Badger: found this in some other posts you did. geoip { source =\> "[fwStuff][src]" } Many thanks to all! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 22, 2019, 8:42pm UTC](https://discuss.elastic.co/t/grok-firewall-logs/200749/8 "2019-10-22T20:42:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.