Fetching all external IP address from firewall logs using logstash

Elastic Stack Logstash
1

I am working with fortinet firewall logs, trying to get all external IP address from the fields srcip and dstip into a text file. I am new to writing filters for the logstash.

The sample documents are as follow but I am struggling to write the right filter for outing the ip address

date=2019-05-10 time=11:37:47 logid="0000000013" type="traffic" subtype="forward" level="notice" vd="vdom1" eventtime=1557513467369913239 srcip=10.1.100.11 srcport=58012 srcintf="port12" srcintfrole="undefined" dstip=23.59.154.35 dstport=80 dstintf="port11" dstintfrole="undefined" srcuuid="ae28f494-5735-51e9-f247-d1d2ce663f4b" dstuuid="ae28f494-5735-51e9-f247-d1d2ce663f4b" poluuid="ccb269e0-5735-51e9-a218-a397dd08b7eb" sessionid=105048 proto=6 action="close" policyid=1 policytype="policy" service="HTTP" dstcountry="Canada" srccountry="Reserved" trandisp="snat" transip=172.16.200.2 transport=58012 appid=34050 app="HTTP.BROWSER_Firefox" appcat="Web.Client" apprisk="elevated" applist="g-default" duration=116 sentbyte=1188 rcvdbyte=1224 sentpkt=17 rcvdpkt=16 utmaction="allow" countapp=1 osname="Ubuntu" mastersrcmac="a2:e9:00:ec:40:01" srcmac="a2:e9:00:ec:40:01" srcserver=0 utmref=65500-742


date=2019-05-10 time=14:18:54 logid="0004000017" type="traffic" subtype="sniffer" level="notice" vd="root" eventtime=1557523134021045897 srcip=208.91.114.4 srcport=50463 srcintf="port1" srcintfrole="undefined" dstip=104.80.88.154 dstport=443 dstintf="port1" dstintfrole="undefined" sessionid=2193276 proto=6 action="accept" policyid=3 policytype="sniffer" service="HTTPS" dstcountry="United States" srccountry="Canada" trandisp="snat" transip=0.0.0.0 transport=0 duration=10 sentbyte=0 rcvdbyte=0 sentpkt=0 rcvdpkt=0 appcat="unscanned" utmaction="allow" countips=1 crscore=5 craction=32768 sentdelta=0 rcvddelta=0 utmref=65162-7772

I need a condition too as if the srcip or dstip are not private addresses, then the output IP.txt should contain

23.59.154.35 
208.91.114.4 
104.80.88.154
2

If you use

    kv {}
    prune { whitelist_names => [ "srcip", "dstip" ] }

    cidr { address => [ "%{srcip}" ] network => [ "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16" ] add_tag => [ "srclocal" ] }
    if "srclocal" in [tags] { mutate { remove_field => "srcip" remove_tag => "srclocal" } }

    cidr { address => [ "%{dstip}" ] network => [ "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16" ] add_tag => [ "dstlocal" ] }
    if "dstlocal" in [tags] { mutate { remove_field => "dstip" remove_tag => "dstlocal" } }

then the events will look like

{
    "dstip" => "23.59.154.35",
     "tags" => []
}
{
    "dstip" => "104.80.88.154",
    "srcip" => "208.91.114.4"
}

You can then extract just the IP addresses using

    clone { clones => "src" }
    if "src" in [tags] {
        prune { whitelist_names => [ "srcip" ] }
        mutate { rename => { "srcip" => "ip" } }
    } else {
        prune { whitelist_names => [ "dstip" ] }
        mutate { rename => { "dstip" => "ip" } }
    }

then write them to a file using

output { if [ip] { file { path => "/tmp/IP.txt" codec => line { format => "%{ip}" } } } }
2 Likes
3

Sorry for the delay @Badger . I have made a configuration with the lines you gave me. I am getting the input from a cloned configuration type called "externaliplist". But when I apply the script, its not giving any output in the output file as its empty. I can't find any errors in the configuration file as it passed the logstash configuration test, but I suspect something within the script not outputting correctly. Can you please have a look in to the following configuration script?

filter {
  if [type] == "externaliplist" {
    kv {}

    prune { whitelist_names => [ "srcip", "dstip" ] }

    cidr { address => [ "%{srcip}" ] network => [ "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16" ] add_tag => [ "srclocal" ] }

    if "srclocal" in [tags] { mutate { remove_field => "srcip" remove_tag => "srclocal" } }

    cidr { address => [ "%{dstip}" ] network => [ "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16" ] add_tag => [ "dstlocal" ] }

    if "dstlocal" in [tags] { mutate { remove_field => "dstip" remove_tag => "dstlocal" } }

    clone { clones => "src" }
    if "src" in [tags] {
        prune { whitelist_names => [ "srcip" ] }
        mutate { rename => { "srcip" => "ip" } }
    } else {
        prune { whitelist_names => [ "dstip" ] }
        mutate { rename => { "dstip" => "ip" } }
    }
  }
}

output {
   if [ip] { file
             {
             path  => "/var/log/logstash/ips"
             codec => line { format => "%{ip}" }
             }
   }
}

for you reference, I tried output the type "externaliplist" without any filter and in realtime and the following is the live sample lines.

2023-06-06T12:43:33.093Z 192.168.2.24 <190>date=2023-06-06 time=13:43:33 devname="XXXXXX" devid="XXXXX" eventtime=1686055412985782444 tz="+0100" logid="1059028704" type="utm" subtype="app-ctrl" eventtype="signature" level="information" vd="root" appid=43541 srcip=106.1.24.54 dstip=72.102.74.7 srcport=49846 dstport=443 srcintf="port1" srcintfrole="lan" dstintf="port2" dstintfrole="wan" proto=6 service="SSL" direction="outgoing" policyid=72 sessionid=3326695785 applist="Allow-Policy" action="pass" appcat="Collaboration" app="Microsoft.Teams" hostname="teams.microsoft.com" incidentserialno=2064343247 url="/" msg="Collaboration: Microsoft.Teams," apprisk="elevated" scertcname="teams.microsoft.com" scertissuer="TLS Issuing CA 06"
4

When I apply those filter to that log entry to get two events with an [ip] field. I do not know what to suggest.

5

I think first part of the log causing the issue, any idea how to remove these with a pattern from the following logs before applying the those filter?

2023-06-06T12:43:33.093Z 192.168.2.24 <190>date=2023-06-06 time=13:43:33 devname="XXXXXX" devid="XXXXX" eventtime=1686055412985782444 tz="+0100" logid="1059028704" type="utm" subtype="app-ctrl" eventtype="signature" level="information" vd="root" appid=43541 srcip=106.1.24.54 dstip=72.102.74.7 srcport=49846 dstport=443 srcintf="port1" srcintfrole="lan" dstintf="port2" dstintfrole="wan" proto=6 service="SSL" direction="outgoing" policyid=72 sessionid=3326695785 applist="Allow-Policy" action="pass" appcat="Collaboration" app="Microsoft.Teams" hostname="teams.microsoft.com" incidentserialno=2064343247 url="/" msg="Collaboration: Microsoft.Teams," apprisk="elevated" scertcname="teams.microsoft.com" scertissuer="TLS Issuing CA 06"
6

You could use dissect

dissect { mapping => { "message" => "%{[@metadata][ts]} %{} <%{}>%{[@metadata][restOfLine]}" } }

then use the kv filter with [@metadata][restOfLine] as the source.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.