# Fetching all external IP address from firewall logs using logstash **URL:** \ **Category:** Logstash\ **Created:** [June 1, 2023, 9:14am UTC](https://discuss.elastic.co/t/fetching-all-external-ip-address-from-firewall-logs-using-logstash/334934 "2023-06-01T09:14:42Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![libinmath](https://avatars.discourse-cdn.com/v4/letter/l/46a35a/32.png) [@libinmath](https://discuss.elastic.co/u/libinmath)\ **Post date:** [June 1, 2023, 9:14am UTC](https://discuss.elastic.co/t/fetching-all-external-ip-address-from-firewall-logs-using-logstash/334934/1 "2023-06-01T09:14:42Z")
I am working with fortinet firewall logs, trying to get all external IP address from the fields srcip and dstip into a text file. I am new to writing filters for the logstash. The sample documents are as follow but I am struggling to write the right filter for outing the ip address ```auto date=2019-05-10 time=11:37:47 logid="0000000013" type="traffic" subtype="forward" level="notice" vd="vdom1" eventtime=1557513467369913239 srcip=10.1.100.11 srcport=58012 srcintf="port12" srcintfrole="undefined" dstip=23.59.154.35 dstport=80 dstintf="port11" dstintfrole="undefined" srcuuid="ae28f494-5735-51e9-f247-d1d2ce663f4b" dstuuid="ae28f494-5735-51e9-f247-d1d2ce663f4b" poluuid="ccb269e0-5735-51e9-a218-a397dd08b7eb" sessionid=105048 proto=6 action="close" policyid=1 policytype="policy" service="HTTP" dstcountry="Canada" srccountry="Reserved" trandisp="snat" transip=172.16.200.2 transport=58012 appid=34050 app="HTTP.BROWSER_Firefox" appcat="Web.Client" apprisk="elevated" applist="g-default" duration=116 sentbyte=1188 rcvdbyte=1224 sentpkt=17 rcvdpkt=16 utmaction="allow" countapp=1 osname="Ubuntu" mastersrcmac="a2:e9:00:ec:40:01" srcmac="a2:e9:00:ec:40:01" srcserver=0 utmref=65500-742 date=2019-05-10 time=14:18:54 logid="0004000017" type="traffic" subtype="sniffer" level="notice" vd="root" eventtime=1557523134021045897 srcip=208.91.114.4 srcport=50463 srcintf="port1" srcintfrole="undefined" dstip=104.80.88.154 dstport=443 dstintf="port1" dstintfrole="undefined" sessionid=2193276 proto=6 action="accept" policyid=3 policytype="sniffer" service="HTTPS" dstcountry="United States" srccountry="Canada" trandisp="snat" transip=0.0.0.0 transport=0 duration=10 sentbyte=0 rcvdbyte=0 sentpkt=0 rcvdpkt=0 appcat="unscanned" utmaction="allow" countips=1 crscore=5 craction=32768 sentdelta=0 rcvddelta=0 utmref=65162-7772 ``` I need a condition too as if the srcip or dstip are not private addresses, then the output IP.txt should contain ```auto 23.59.154.35 208.91.114.4 104.80.88.154 ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 1, 2023, 7:42pm UTC](https://discuss.elastic.co/t/fetching-all-external-ip-address-from-firewall-logs-using-logstash/334934/2 "2023-06-01T19:42:30Z")
If you use ``` kv {} prune { whitelist_names => ["srcip", "dstip"] } cidr { address => ["%{srcip}"] network => ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"] add_tag => ["srclocal"] } if "srclocal" in [tags] { mutate { remove_field => "srcip" remove_tag => "srclocal" } } cidr { address => ["%{dstip}"] network => ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"] add_tag => ["dstlocal"] } if "dstlocal" in [tags] { mutate { remove_field => "dstip" remove_tag => "dstlocal" } } ``` then the events will look like ``` { "dstip" => "23.59.154.35", "tags" => [] } { "dstip" => "104.80.88.154", "srcip" => "208.91.114.4" } ``` You can then extract just the IP addresses using ``` clone { clones => "src" } if "src" in [tags] { prune { whitelist_names => ["srcip"] } mutate { rename => { "srcip" => "ip" } } } else { prune { whitelist_names => ["dstip"] } mutate { rename => { "dstip" => "ip" } } } ``` then write them to a file using ``` output { if [ip] { file { path => "/tmp/IP.txt" codec => line { format => "%{ip}" } } } } ``` ---
**Author:** ![libinmath](https://avatars.discourse-cdn.com/v4/letter/l/46a35a/32.png) [@libinmath](https://discuss.elastic.co/u/libinmath)\ **Post date:** [June 6, 2023, 12:29pm UTC](https://discuss.elastic.co/t/fetching-all-external-ip-address-from-firewall-logs-using-logstash/334934/3 "2023-06-06T12:29:30Z")
Sorry for the delay @Badger . I have made a configuration with the lines you gave me. I am getting the input from a cloned configuration type called "externaliplist". But when I apply the script, its not giving any output in the output file as its empty. I can't find any errors in the configuration file as it passed the logstash configuration test, but I suspect something within the script not outputting correctly. Can you please have a look in to the following configuration script? ```auto filter { if [type] == "externaliplist" { kv {} prune { whitelist_names => ["srcip", "dstip"] } cidr { address => ["%{srcip}"] network => ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"] add_tag => ["srclocal"] } if "srclocal" in [tags] { mutate { remove_field => "srcip" remove_tag => "srclocal" } } cidr { address => ["%{dstip}"] network => ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"] add_tag => ["dstlocal"] } if "dstlocal" in [tags] { mutate { remove_field => "dstip" remove_tag => "dstlocal" } } clone { clones => "src" } if "src" in [tags] { prune { whitelist_names => ["srcip"] } mutate { rename => { "srcip" => "ip" } } } else { prune { whitelist_names => ["dstip"] } mutate { rename => { "dstip" => "ip" } } } } } output { if [ip] { file { path => "/var/log/logstash/ips" codec => line { format => "%{ip}" } } } } ``` for you reference, I tried output the type "externaliplist" without any filter and in realtime and the following is the live sample lines. ```auto 2023-06-06T12:43:33.093Z 192.168.2.24 <190>date=2023-06-06 time=13:43:33 devname="XXXXXX" devid="XXXXX" eventtime=1686055412985782444 tz="+0100" logid="1059028704" type="utm" subtype="app-ctrl" eventtype="signature" level="information" vd="root" appid=43541 srcip=106.1.24.54 dstip=72.102.74.7 srcport=49846 dstport=443 srcintf="port1" srcintfrole="lan" dstintf="port2" dstintfrole="wan" proto=6 service="SSL" direction="outgoing" policyid=72 sessionid=3326695785 applist="Allow-Policy" action="pass" appcat="Collaboration" app="Microsoft.Teams" hostname="teams.microsoft.com" incidentserialno=2064343247 url="/" msg="Collaboration: Microsoft.Teams," apprisk="elevated" scertcname="teams.microsoft.com" scertissuer="TLS Issuing CA 06" ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 6, 2023, 7:01pm UTC](https://discuss.elastic.co/t/fetching-all-external-ip-address-from-firewall-logs-using-logstash/334934/4 "2023-06-06T19:01:38Z")
When I apply those filter to that log entry to get two events with an [ip] field. I do not know what to suggest. ---
**Author:** ![libinmath](https://avatars.discourse-cdn.com/v4/letter/l/46a35a/32.png) [@libinmath](https://discuss.elastic.co/u/libinmath)\ **Post date:** [June 7, 2023, 8:25am UTC](https://discuss.elastic.co/t/fetching-all-external-ip-address-from-firewall-logs-using-logstash/334934/5 "2023-06-07T08:25:50Z")
> [@libinmath](#): > > `2023-06-06T12:43:33.093Z 192.168.2.24 <190>` I think first part of the log causing the issue, any idea how to remove these with a pattern from the following logs before applying the those filter? ```auto 2023-06-06T12:43:33.093Z 192.168.2.24 <190>date=2023-06-06 time=13:43:33 devname="XXXXXX" devid="XXXXX" eventtime=1686055412985782444 tz="+0100" logid="1059028704" type="utm" subtype="app-ctrl" eventtype="signature" level="information" vd="root" appid=43541 srcip=106.1.24.54 dstip=72.102.74.7 srcport=49846 dstport=443 srcintf="port1" srcintfrole="lan" dstintf="port2" dstintfrole="wan" proto=6 service="SSL" direction="outgoing" policyid=72 sessionid=3326695785 applist="Allow-Policy" action="pass" appcat="Collaboration" app="Microsoft.Teams" hostname="teams.microsoft.com" incidentserialno=2064343247 url="/" msg="Collaboration: Microsoft.Teams," apprisk="elevated" scertcname="teams.microsoft.com" scertissuer="TLS Issuing CA 06" ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 7, 2023, 4:27pm UTC](https://discuss.elastic.co/t/fetching-all-external-ip-address-from-firewall-logs-using-logstash/334934/6 "2023-06-07T16:27:27Z")
You could use dissect ``` dissect { mapping => { "message" => "%{[@metadata][ts]} %{} <%{}>%{[@metadata][restOfLine]}" } } ``` then use the kv filter with [@metadata][restOfLine] as the source. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2023, 4:27pm UTC](https://discuss.elastic.co/t/fetching-all-external-ip-address-from-firewall-logs-using-logstash/334934/7 "2023-07-05T16:27:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.