Problem With parsing mikrotik log

asan · February 21, 2021, 8:58am

Hi
i've been trying to normalize mikrotik log with grok in logstash and my sample log is :

Feb 20 04:38:02 192.168.202.101 id=firewall sn=C0EAE45CA55 time="2021-02-20 12:54:43" fw=188.126.145.3 pri=6 c=1 m=911 msg="Added host entry to dynamic address object" n=6776584 note="FQDN=*.microsoft.com; TTL=56; Host=20.49.150.241" fw_action="NA"

and this is what i did up to now :

%{SYSLOGTIMESTAMP:syslog_timestamp} %{IPV4:sensor_ip} %{DATA:id}=%{WORD:device_name} %{DATA:sn}=%{USERNAME:device_id} time="%{TIMESTAMP_ISO8601:device_time}"

this parsing all fields except this part time="%{TIMESTAMP_ISO8601:device_time}" and my logstash faced with problem and failed i would be thankful if someone help me to normalize this log.

ylasri · February 21, 2021, 10:20am

Use this grok to parse the message

%{SYSLOGTIMESTAMP:syslog_timestamp} %{IPV4:sensor_ip} %{GREEDYDATA:kv_message}

combined with a KV filter to get dynamically all fields, refer to this issue on github also to get an idea

   kv {
       source => "kv_message"
       "field_split" : """\s(?![-_.,:()\w ]+?(\s+|$))""",
       "value_split" : """(?<!\\)=""",
   }

asan · February 23, 2021, 5:17am

thanks Bro,
i fixed my problem and i'd like to share this link with others for normalizing mikrotik logs.

github.com

frap/logstash/blob/master/patterns/mikrotik

# Notes: ?: => Non-capturing group (http://docs.python.org/howto/regex)
#  ? => Match zero or one of the preceding
#  | => Pattern separate - logical OR
#  - => Literal hyphen

MTIKBSDSYSLOG  ^<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{WORD:log_source} %{GREEDYDATA:syslog_message}


# Firewall Log (No NAT):
MIKROTIKFIREWALLNONAT %{DATA:LogPrefix} %{DATA:LogChain}: in:%{DATA:src_zone} out:%{DATA:dst_zone}, src-mac %{MAC}, proto %{DATA:proto}, %{IP:src_ip}:%{INT:src_port}->%{IP:dst_ip}:%{INT:dst_port}, len %{INT:length}

# Firewall Log (NAT):
MIKROTIKFIREWALLNAT %{DATA:LogPrefix} %{DATA:LogChain}: in:%{DATA:src_zone} out:%{DATA:dst_zone}, src-mac %{MAC}, proto %{DATA:proto}, %{IP:src_ip}:%{INT:src_port}->%{IP:dst_ip}:%{INT:dst_port}, NAT \(%{IP:nat_osrc_ip}:%{INT:nat_osrc_port}->%{IP:nat_nsrc_ip}:%{INT:nat_nsrc_port}\)->%{IP:nat_dst_ip}:%{INT:nat_dstport}, len %{INT:length}
MIKROTIKFIREWALLNAT2 %{DATA:LogPrefix} %{DATA:LogChain}: in:%{DATA:src_zone} out:%{DATA:dst_zone}, proto %{DATA:proto}, %{IP:src_ip}:%{INT:src_port}->%{IP:dst_ip}:%{INT:dst_port}, NAT %{IP:nat_osrc_ip}:%{INT:nat_osrc_port}->\(%{IP:nat_nsrc_ip}:%{INT:nat_nsrc_port}->%{IP:nat_dst_ip}:%{INT:nat_dstport}\), len %{INT:length}

MIKROTIKFIREWALL (?:%{MIKROTIKFIREWALLNAT}|%{MIKROTIKFIREWALLNAT2}|%{MIKROTIKFIREWALLNONAT})

# DNS
MIKROTIKDNSQUERY  query from %{IP:src_ip}: #%{INT:query_id} %{GREEDYDATA:query}
MIKROTIKDNSANSWER  done query: #%{INT:query_id} %{GREEDYDATA:query}

This file has been truncated. show original

system · March 23, 2021, 5:17am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.