Split multiple CGN Logs from Cisco Router to separate Documents

[neerajsapkota (ABCXYZ)]

Here is the syslog message from Cisco Router..
I get this output in Kibana Discover tab in message field.

<134> 1 2018 Aug 10 05:48:17 - - NAT44 - [SessionbasedWD 17 10.10.10.2 poolA - 192.168.1.2 40945 41011 - 172.16.1.2 53 ][SessionbasedWD 6 10.10.10.5 poolA - 192.168.1.2 43752 6182 - 172.16.1.5 443 ][SessionbasedWD 17 10.10.10.16 poolA - 192.168.1.3 47766 33538 - 172.16.1.10 8615 ][SessionbasedWD 6 10.10.10.17 poolA - 192.168.1.3 2803 7171 - 172.16.1.15 80 ][SessionbasedWD 6 10.10.10.19 poolA - 192.168.1.3 15523 53551 - 172.16.1.54 443 ][SessionbasedWD 17 10.10.10.10 poolA - 192.168.1.2 14907 24773 - 172.16.1.38 53 ][SessionbasedAD 17 10.10.10.11 poolA - 192.168.1.5 45145 26295 - 172.16.1.26 53 ][SessionbasedAD 17 10.10.10.24 poolA - 192.168.1.10 45145 26295 - 172.16.1.87 53 ][SessionbasedWD 17 10.10.10.222 poolA - 192.168.1.30 51503 32237 - 172.16.1.90 53 ]

I have used this grok filter to parse the log:

message : match => { "message" => "\s1\s%{NUMBER:year}\s%{WORD:month}\s%{NUMBER:day}\s%{TIME:time}\s\s-\s-\s%{WORD:CGN_type}\s-\s[%{WORD:session_type}\s%{NUMBER:protocol}\s%{IP:src_ip_private}\s%{WORD:inside_vrf}\s-\s%{IP:src_ip_public_translated}\s%{NUMBER:src_port}\s%{NUMBER:src_port_translated}\s-\s%{IP:dst_ip}\s%{NUMBER:dst_port}\s]" }

Grok parses the syslog successfully but only the first occurrence is matched and rest of the information is supressed.

Now, I need to Split the syslog into separate documents so that the message field in each document is as follows:

document1: <134> 1 2018 Aug 10 05:48:17 - - NAT44 - [SessionbasedWD 17 10.10.10.2 poolA - 192.168.1.2 40945 41011 - 172.16.1.2 53 ]

document2: <134> 1 2018 Aug 10 05:48:17 - - NAT44 - [SessionbasedWD 6 10.10.10.5 poolA - 192.168.1.2 43752 6182 - 172.16.1.5 443 ]

document3: <134> 1 2018 Aug 10 05:48:17 - - NAT44 - [SessionbasedWD 17 10.10.10.16 poolA - 192.168.1.3 47766 33538 - 172.16.1.10 8615 ]

and so on.

Any help will be Appreciated.

[magnusbaeck (Magnus Bäck)]

You'll want to create an array with all the [SessionbasedWD ...] strings, i.e. your event should look something like

{
  "@timestamp": ...,
  "entries": [
    "[SessionbasedWD 17 10.10.10.2 poolA - 192.168.1.2 40945 41011 - 172.16.1.2 53 ]",
    "[SessionbasedWD 6 10.10.10.5 poolA - 192.168.1.2 43752 6182 - 172.16.1.5 443 ]",
    "[SessionbasedWD 17 10.10.10.16 poolA - 192.168.1.3 47766 33538 - 172.16.1.10 8615 ]",
    "[SessionbasedWD 6 10.10.10.17 poolA - 192.168.1.3 2803 7171 - 172.16.1.15 80 ]",
    ...
  ],
  ...
}

and then you use a split filter on that array field. I think you'll be able to apply further filtering (like grok) on the resulting events. To create the array you should be able to use a mutate filter and its split option (not to be confused with the split filter) and split the string on ][.

[neerajsapkota (ABCXYZ)]

Thanks a lot @magnusbaeck . I got it working with the config below:

input {
   udp {
    port => 5544
        type => "syslog"
        }
      }

filter {
  if [type] == "syslog" {
    grok {
match => { "message" => "(?<info>[^\[]*)(?<patterns>[^\$]*)" }
remove_field => ["message"] 
    }
mutate {
        split => { "patterns" => "]["}
    }
 split{
        field => "patterns"
    }
  }
}

output {
if [type] == "syslog" {
  elasticsearch {
    hosts => ["localhost:9200"]
                }
}
}