asan
(asan)
February 21, 2021, 8:58am
1
Hi
i've been trying to normalize mikrotik log with grok in logstash and my sample log is :
Feb 20 04:38:02 192.168.202.101 id=firewall sn=C0EAE45CA55 time="2021-02-20 12:54:43" fw=188.126.145.3 pri=6 c=1 m=911 msg="Added host entry to dynamic address object" n=6776584 note="FQDN=*.microsoft.com; TTL=56; Host=20.49.150.241" fw_action="NA"
and this is what i did up to now :
%{SYSLOGTIMESTAMP:syslog_timestamp} %{IPV4:sensor_ip} %{DATA:id}=%{WORD:device_name} %{DATA:sn}=%{USERNAME:device_id} time="%{TIMESTAMP_ISO8601:device_time}"
this parsing all fields except this part time="%{TIMESTAMP_ISO8601:device_time}" and my logstash faced with problem and failed i would be thankful if someone help me to normalize this log.
ylasri
(Yassine LASRI)
February 21, 2021, 10:17am
2
Use this grok to parse the message
%{SYSLOGTIMESTAMP:syslog_timestamp} %{IPV4:sensor_ip} %{GREEDYDATA:kv_message}
combined with a KV filter to get dynamically all fields, refer to this issue on github also to get an idea
kv {
source => "kv_message"
"field_split" : """\s(?![-_.,:()\w ]+?(\s+|$))""",
"value_split" : """(?<!\\)=""",
}
1 Like
asan
(asan)
February 23, 2021, 5:17am
3
thanks Bro,
i fixed my problem and i'd like to share this link with others for normalizing mikrotik logs.
# Notes: ?: => Non-capturing group (http://docs.python.org/howto/regex)
# ? => Match zero or one of the preceding
# | => Pattern separate - logical OR
# - => Literal hyphen
MTIKBSDSYSLOG ^<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{WORD:log_source} %{GREEDYDATA:syslog_message}
# Firewall Log (No NAT):
MIKROTIKFIREWALLNONAT %{DATA:LogPrefix} %{DATA:LogChain}: in:%{DATA:src_zone} out:%{DATA:dst_zone}, src-mac %{MAC}, proto %{DATA:proto}, %{IP:src_ip}:%{INT:src_port}->%{IP:dst_ip}:%{INT:dst_port}, len %{INT:length}
# Firewall Log (NAT):
MIKROTIKFIREWALLNAT %{DATA:LogPrefix} %{DATA:LogChain}: in:%{DATA:src_zone} out:%{DATA:dst_zone}, src-mac %{MAC}, proto %{DATA:proto}, %{IP:src_ip}:%{INT:src_port}->%{IP:dst_ip}:%{INT:dst_port}, NAT \(%{IP:nat_osrc_ip}:%{INT:nat_osrc_port}->%{IP:nat_nsrc_ip}:%{INT:nat_nsrc_port}\)->%{IP:nat_dst_ip}:%{INT:nat_dstport}, len %{INT:length}
MIKROTIKFIREWALLNAT2 %{DATA:LogPrefix} %{DATA:LogChain}: in:%{DATA:src_zone} out:%{DATA:dst_zone}, proto %{DATA:proto}, %{IP:src_ip}:%{INT:src_port}->%{IP:dst_ip}:%{INT:dst_port}, NAT %{IP:nat_osrc_ip}:%{INT:nat_osrc_port}->\(%{IP:nat_nsrc_ip}:%{INT:nat_nsrc_port}->%{IP:nat_dst_ip}:%{INT:nat_dstport}\), len %{INT:length}
MIKROTIKFIREWALL (?:%{MIKROTIKFIREWALLNAT}|%{MIKROTIKFIREWALLNAT2}|%{MIKROTIKFIREWALLNONAT})
# DNS
MIKROTIKDNSQUERY query from %{IP:src_ip}: #%{INT:query_id} %{GREEDYDATA:query}
MIKROTIKDNSANSWER done query: #%{INT:query_id} %{GREEDYDATA:query}
This file has been truncated. show original
system
(system)
Closed
March 23, 2021, 5:17am
4
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.