Grok failure, Grok debug OK

[Abdelhalim]

Hello,

I am using grok to parse my logs, everything works in grok debugger in Kibana but when I use it in my pipeline it doesn't work!!

here is an example of my log:

Jul 10 12:21:08 vselk-vyos kernel: [155099.389432] [ELK-IN-1-A] IN=eth1 OUT=eth0 MAC=00:0c:29:c1:78:1b:00:0c:29:b4:a3:af:08:00 SRC=10.13.81.30 DST=10.10.13.135 LEN=52 TOS=0x10 PREC=0x00 TTL=63 ID=62337 DF PROTO=TCP SPT=22 DPT=54140 WINDOW=501 RES=0x00 ACK URGP=0

and here is my pipeline:

filter {
grok {
match => {
"message" => [
"%{SYSLOGTIMESTAMP:nf_timestamp} %{HOSTNAME:nf_host} kernel: [%{INT:port_srouce}.%{INT:sourceccc}] [%{DATA:iptable_rule}] IN=%{WORD:Interface_entrer} OUT=%{WORD:interface_sortie} MAC=%{DATA:MAC} SRC=%{IP:IP_source} DST=%{IP:IP_Dest} LEN=%{NUMBER:Pakcet_Length} TOS=%{DATA:Type_Of_Service} PREC=%{DATA:Precedent_bits} TTL=%{NUMBER:Time_To_Live} ID=%{NUMBER:Packet_identifier} PROTO=%{WORD:Protocol} TYPE=%{NUMBER:Type} CODE=%{NUMBER:Code} ID=%{NUMBER:ID} SEQ=%{NUMBER:Sequence_number}",
"%{SYSLOGTIMESTAMP:nf_timestamp} %{HOSTNAME:nf_host} kernel: [%{INT:port_srouce}.%{INT:sourceccc}] [%{DATA:iptable_rule}] IN=%{WORD:Interface_entrer} OUT=%{WORD:interface_sortie} MAC=%{DATA:MAC} SRC=%{IP:IP_source} DST=%{IP:IP_Dest} LEN=%{NUMBER:Pakcet_Length} TOS=%{DATA:Type_Of_Service} PREC=%{DATA:Precedent_bits} TTL=%{NUMBER:Time_To_Live} ID=%{NUMBER:Packet_identifier} %{WORD:Fragment_Flag} PROTO=%{WORD:Protocol} TYPE=%{NUMBER:Type} CODE=%{NUMBER:Code} ID=%{NUMBER:ID} SEQ=%{NUMBER:Sequence_number}",
"%{SYSLOGTIMESTAMP:nf_timestamp} %{HOSTNAME:nf_host} kernel: [%{INT:port_srouce}.%{INT:sourceccc}] [%{DATA:iptable_rule}] IN=%{WORD:Interface_entrer} OUT=%{WORD:interface_sortie} MAC=%{DATA:MAC} SRC=%{IP:IP_source} DST=%{IP:IP_Dest} LEN=%{NUMBER:Pakcet_Length} TOS=%{DATA:Type_Of_Service} PREC=%{DATA:Precedent_bits} TTL=%{NUMBER:Time_To_Live} ID=%{NUMBER:Packet_identifier} %{WORD:fragment_flag} PROTO=%{WORD:Protocol} SPT=%{NUMBER=source_port} DPT=%{NUMBER:destination_Port} WINDOW=%{NUMBER:Window_size} RES=%{WORD:Reserved_bytes} %{DATA:Connexion_type} URGP=%{NUMBER:Urgent_packet}"]
}
}
}

the line in strong is the line which is no working, could someone help me please !

Thank you

[Jenni]

For that grok pattern I would suggest escaping the brackets and period as well as correcting the NUMBER=source_port:

%{SYSLOGTIMESTAMP:nf_timestamp} %{HOSTNAME:nf_host} kernel: \[%{INT:port_srouce}\.%{INT:sourceccc}\] \[%{DATA:iptable_rule}\] IN=%{WORD:Interface_entrer} OUT=%{WORD:interface_sortie} MAC=%{DATA:MAC} SRC=%{IP:IP_source} DST=%{IP:IP_Dest} LEN=%{NUMBER:Pakcet_Length} TOS=%{DATA:Type_Of_Service} PREC=%{DATA:Precedent_bits} TTL=%{NUMBER:Time_To_Live} ID=%{NUMBER:Packet_identifier} %{WORD:fragment_flag} PROTO=%{WORD:Protocol} SPT=%{NUMBER:source_port} DPT=%{NUMBER:destination_Port} WINDOW=%{NUMBER:Window_size} RES=%{WORD:Reserved_bytes} %{DATA:Connexion_type} URGP=%{NUMBER:Urgent_packet}

But further more I'd like to say that

• You've got some typos in the field names you chose.
• It would probably make it a lot easier to keep an overview if you only used grok for the first part of the log line and a KV filter for the rest and renamed the fields afterwards.
  %{SYSLOGTIMESTAMP:nf_timestamp} %{HOSTNAME:nf_host} kernel: \[%{INT:port_srouce}\.%{INT:sourceccc}\] \[%{DATA:iptable_rule}\] %{GREEDYDATA:rest}