Hello, I have two different time format in my logs like "Aug 10 09:18:10" and "2020-08-10T08:59:59.415319+02:00" I tried many different ways to have them in one grok but I fail date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss", "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZZ" ]
when I use "syslog_timestamp", "MMM d HH:mm:ss" my filtering works but for the other type, it doesn't work. Is there anybody had the same issue? any suggestion? Thank you in advance
(You irritated me at first because you talk about grok, even though this has nothing to do with a grok filter.)
I don't see any problem with your current patterns, but you could simplify this by using "ISO8601" for your second format.
1 Like
For the beginning of your grok pattern you can use (?<syslog_timestamp>%{SYSLOGTIMESTAMP}|%{TIMESTAMP_ISO8601})
And as a side note: You don't have to give every part of your grok a name. If you used %{GREEDYDATA} instead of %{GREEDYDATA:vpn_junk_1} there wouldn't be a field that you have to delete later on.
1 Like
Could you post an example?
My minimalistic test works fine.
input{
stdin{}
}
filter {
grok {
match => ["message", "(?<syslog_timestamp>%{SYSLOGTIMESTAMP}|%{TIMESTAMP_ISO8601}) (?<msg>.*)" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "ISO8601" ]
}
}
output{
stdout{}
}
The stdin plugin is now waiting for input:
2020-08-10T08:59:59.415319+02:00 oh
{
"message" => "2020-08-10T08:59:59.415319+02:00 oh",
"@version" => "1",
"host" => "#####",
"msg" => "oh",
"syslog_timestamp" => "2020-08-10T08:59:59.415319+02:00",
"@timestamp" => 2020-08-10T06:59:59.415Z
}
Aug 10 09:18:10 ah
{
"message" => "Aug 10 09:18:10 ah",
"@version" => "1",
"host" => "#####",
"msg" => "ah",
"syslog_timestamp" => "Aug 10 09:18:10",
"@timestamp" => 2020-08-10T07:18:10.000Z
}
I'll have a closer look at that a bit later. But that log looks like a perfect use case for a KV filter 
1 Like
input{
stdin{}
}
filter {
grok {
match => ["message", "(?<syslog_timestamp>%{SYSLOGTIMESTAMP}|%{TIMESTAMP_ISO8601}) %{SYSLOGHOST:syslog_hostname} %{GREEDYDATA:kvdata}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "ISO8601" ]
}
kv {
source => "kvdata"
field_split => "|"
include_keys => ["office_mode_ip", "src", "user", "user_group"]
prefix => "vpn_"
}
}
output{
stdout{}
}
The stdin plugin is now waiting for input:
Aug 10 11:32:10 charlie.xxx.com time=1596443329|hostname=charlie|product=Mobile Access|action=Log In|ifdir=inbound|loguid={0x5f27xsc1,0x1,0x1df7b62e,0x342d}|origin=0.0.0.0|originsicname=CN=asa01,O=charlie.xxx.com.ea5xhn|sequencenum=300|version=5|auth_encryption_methods=AES-256 + SHA1 + Group 2|auth_method=Password|client_build=964101311|client_name=Endpoint Security |client_version=E820|cvpn_category=Session|device_identification={8981345E-48C3-4E63-A27D-83058E43F12F}|domain_name=lara.com|event_type=Login|failed_login_factor_num=0|host_ip=0.0.0.0|host_type=PC|hostname=SC|lastupdatetime=1594332230|login_option=Standard|login_timestamp=1596423329|mac_address=10:20:56:94:2c:b1|office_mode_ip=0.0.0.0|os_bits=32bit|os_build=17563|os_edition=Professional|os_name=Windows|os_version=10|proto=6|proxy_src_ip=0.0.0.0|s_port=0|service=443|session_timeout=28700|session_uid={5as7ASC1-1000-0000-2CB6-F71D2D340000}|src=0.0.0.0|status=Success|suppressed_logs=0|tunnel_protocol=IPSec|user=barbara|user_dn=barbara|user_group=Support_Javad|"
{
"@timestamp" => 2020-08-10T09:32:10.000Z,
"@version" => "1",
"message" => "Aug 10 11:32:10 charlie.xxx.com time=1596443329|hostname=charlie|product=Mobile Access|action=Log In|ifdir=inbound|loguid={0x5f27xsc1,0x1,0x1df7b62e,0x342d}|origin=0.0.0.0|originsicname=CN=asa01,O=charlie.xxx.com.ea5xhn|sequencenum=300|version=5|auth_encryption_methods=AES-256 + SHA1 + Group 2|auth_method=Password|client_build=964101311|client_name=Endpoint Security |client_version=E820|cvpn_category=Session|device_identification={8981345E-48C3-4E63-A27D-83058E43F12F}|domain_name=lara.com|event_type=Login|failed_login_factor_num=0|host_ip=0.0.0.0|host_type=PC|hostname=SC|lastupdatetime=1594332230|login_option=Standard|login_timestamp=1596423329|mac_address=10:20:56:94:2c:b1|office_mode_ip=0.0.0.0|os_bits=32bit|os_build=17563|os_edition=Professional|os_name=Windows|os_version=10|proto=6|proxy_src_ip=0.0.0.0|s_port=0|service=443|session_timeout=28700|session_uid={5as7ASC1-1000-0000-2CB6-F71D2D340000}|src=0.0.0.0|status=Success|suppressed_logs=0|tunnel_protocol=IPSec|user=barbara|user_dn=barbara|user_group=Support_Javad|\"",
"syslog_hostname" => "charlie.xxx.com",
"host" => "#####",
"vpn_user_group" => "Support_Javad",
"vpn_office_mode_ip" => "0.0.0.0",
"kvdata" => "time=1596443329|hostname=charlie|product=Mobile Access|action=Log In|ifdir=inbound|loguid={0x5f27xsc1,0x1,0x1df7b62e,0x342d}|origin=0.0.0.0|originsicname=CN=asa01,O=charlie.xxx.com.ea5xhn|sequencenum=300|version=5|auth_encryption_methods=AES-256 + SHA1 + Group 2|auth_method=Password|client_build=964101311|client_name=Endpoint Security |client_version=E820|cvpn_category=Session|device_identification={8981345E-48C3-4E63-A27D-83058E43F12F}|domain_name=lara.com|event_type=Login|failed_login_factor_num=0|host_ip=0.0.0.0|host_type=PC|hostname=SC|lastupdatetime=1594332230|login_option=Standard|login_timestamp=1596423329|mac_address=10:20:56:94:2c:b1|office_mode_ip=0.0.0.0|os_bits=32bit|os_build=17563|os_edition=Professional|os_name=Windows|os_version=10|proto=6|proxy_src_ip=0.0.0.0|s_port=0|service=443|session_timeout=28700|session_uid={5as7ASC1-1000-0000-2CB6-F71D2D340000}|src=0.0.0.0|status=Success|suppressed_logs=0|tunnel_protocol=IPSec|user=barbara|user_dn=barbara|user_group=Support_Javad|\"",
"vpn_user" => "barbara",
"syslog_timestamp" => "Aug 10 11:32:10",
"vpn_src" => "0.0.0.0"
}
As you can see I couldn't reproduce the problem. Do you get an error message or tag? What does the rubydebug output look like?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.