Grok failure grok debugger ok

[Antoine.E (Antoine Espinosa)]

Hello everybody,
I'm currently parsing Cisco logs and I don't understand why my logs are marked as parsed failure but when I go and test it on https://grokdebug.herokuapp.com/ everything is parsing perfectly.

My Logstash config for the parsing is

filter {
if [log][file][path] =~ "Switch" {
mutate {
add_tag => ["Cisco"]
}
}
}
filter {
if "Cisco" in [tags]{
if [message] =~ "192.168.50.15" {
mutate{
add_tag => ["LogPointCisco"]
}
grok {
patterns_dir => ["/etc/logstash/conf.d/patterns"]
match => {"message" => "%{IPV4:IP} (%{NUMBER:log_sequence})?: %{NUMBER}: .%{CISCOTIMESTAMP} (%{TZ:UTC}+%{NUMBER:UTC}): %%{CISCOTAG:ciscotag}: %{GREEDYDATA}"}
}
}
else{
grok {
patterns_dir => ["/etc/logstash/conf.d/patterns"]
match => {"message" => "%{IPV4:IP} (%{NUMBER:log_sequence})?: %{NUMBER}: %{CISCOTIMESTAMP} (%{TZ:UTC}+%{NUMBER:UTC}): %%{CISCOTAG:ciscotag}: %{GREEDYDATA}"}
}
}
}
}
output {
if "Cisco" in [tags] {
elasticsearch {
index => "cisco-%{+YYYY.MM.dd}"
hosts => ["http://192.168.50.12:9200"]
}
}
}

The custom patterns that I use is :

CISCOTAG (?<=%).*?(?=:)

This config is parsing 2 type of logs, on with a dot before the timestamp and one without it.
The only switch which is sending me logs with a dot before as 192.168.50.15 as an IP address so i'm checking that.

An exemple of log with a dot would be :

2019-06-27T18:01:59.050870+02:00 172.172.172.2 76977: 076975: .Jun 27 2019 18:01:59.257 UTC+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/13, changed state to up

and one without the dot would be :

2019-06-27T17:44:22.864886+02:00 10.234.125.5 82642: 082796: Jun 27 2019 17:44:21.855 UTC+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to down

I tried to run Logstash in debug mode but no error were displayed.
Thanks in advance ,
Antoine

[Badger]

I do not see any problem with the grok patterns. This parses both lines

input { generator { count => 1 lines => [
'2019-06-27T18:01:59.050870+02:00 172.172.172.2 76977: 076975: .Jun 27 2019 18:01:59.257 UTC+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/13, changed state to up',
'2019-06-27T17:44:22.864886+02:00 10.234.125.5 82642: 082796: Jun 27 2019 17:44:21.855 UTC+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to down' ] } }

filter {
    grok {
        pattern_definitions => { "CISCOTAG" => "(?<=%).*?(?=:)" }
        match => {
            "message" => [
"%{IPV4:IP} (%{NUMBER:log_sequence})?: %{NUMBER}: .%{CISCOTIMESTAMP} (%{TZ:UTC}+%{NUMBER:UTC}): %%{CISCOTAG:ciscotag}: %{GREEDYDATA}",
"%{IPV4:IP} (%{NUMBER:log_sequence})?: %{NUMBER}: %{CISCOTIMESTAMP} (%{TZ:UTC}+%{NUMBER:UTC}): %%{CISCOTAG:ciscotag}: %{GREEDYDATA}"
            ]
        }
    }

}

Although I would change that to use a single pattern with an optional field

"message" => "%{IPV4:IP} (%{NUMBER:log_sequence})?: %{NUMBER}: (.)?%{CISCOTIMESTAMP} (%{TZ:UTC}+%{NUMBER:UTC}): %%{CISCOTAG:ciscotag}: %{GREEDYDATA}"

[Antoine.E (Antoine Espinosa)]

Hello Badger,

The new pattern is perfect and it helped me find the solution,
It looks like the patterns_dir line is not taking into account my custom pattern, I used the pattern_definitions and it's working.

Thanks,
Antoine