Hi im trying to apply the solution described here
to the follwoing part of the Exchange message tracking logs
source_context
MDB:04111c61-c212-4078-bf65-369a8cd3080c, Mailbox:257ff165-97e3-498b-8e11-b5b735da312b, Event:172773384, MessageClass:IPM.Note.MapiSubmitLAMProbe, CreationTime:2018-09-26T09:49:24.396Z, ClientType:Monitoring
to fix the "-" situation in the fields above magnus suggests this line in the grok filter
(-|%{PATTERN:fieldname})
but i can not figure out how to apply this in a grok filter that works
below is my grok that i Thought would work
{WORD:MDB},%{SPACE}%{WORD:Mailbox},%{SPACE}%{INTEGER:Event},%{SPACE}%{WORD:MessageClass},%{SPACE}%{TIMESTAMP_ISO8601:CreationTime},%{SPACE}%{WORD:ClientType}
after findeing the post above i made the following changes
(-|%{WORD:MDB}),%{SPACE}(-|%{WORD:Mailbox}),%{SPACE}%{INTEGER:Event},%{SPACE}%{WORD:MessageClass},%{SPACE}%{TIMESTAMP_ISO8601:CreationTime},%{SPACE}%{WORD:ClientType}
but im still hitting a compile error so it seams i getting the structure of the grok pattern wrong some how.
the entire current filter below
if [type] == "exchange" {
csv {
add_tag => [ 'exh_msg_trk' ]
columns => ['logdate', 'client_ip', 'client_hostname', 'server_ip', 'server_hostname', 'source_context', 'connector_id', 'source', 'event_id', 'internal_message_id', 'message_id', 'network_message_id', 'recipient_address', 'recipient_status', 'total_bytes', 'recipient_count', 'related_recipient_address', 'reference', 'message_subject', 'sender_address', 'return_path', 'message_info', 'directionality', 'tenant_id', 'original_client_ip', 'original_server_ip', 'custom_data']
remove_field => [ "logdate" ]
}
grok {
match => [ "message", "%{TIMESTAMP_ISO8601:timestamp}" ]
}
mutate {
convert => [ "total_bytes", "integer" ]
convert => [ "recipient_count", "integer" ]
split => ["recipient_address", ";"]
split => [ "source_context", ";" ]
split => [ "custom_data", ";" ]
}
grok {
match => [ "source_context", "%{GREEDYDATA}%{WORD:ClientType}" ]
}
date {
match => [ "timestamp", "ISO8601" ]
timezone => "Europe/London"
remove_field => [ "timestamp" ]
}
if "_grokparsefailure" in [tags] {
drop { }
}
}
the working filter should look like this ?
if [type] == "exchange" {
csv {
add_tag => [ 'exh_msg_trk' ]
columns => ['logdate', 'client_ip', 'client_hostname', 'server_ip', 'server_hostname', 'source_context', 'connector_id', 'source', 'event_id', 'internal_message_id', 'message_id', 'network_message_id', 'recipient_address', 'recipient_status', 'total_bytes', 'recipient_count', 'related_recipient_address', 'reference', 'message_subject', 'sender_address', 'return_path', 'message_info', 'directionality', 'tenant_id', 'original_client_ip', 'original_server_ip', 'custom_data']
remove_field => [ "logdate" ]
}
grok {
match => [ "message", "%{TIMESTAMP_ISO8601:timestamp}" ]
}
mutate {
convert => [ "total_bytes", "integer" ]
convert => [ "recipient_count", "integer" ]
split => ["recipient_address", ";"]
split => [ "source_context", ";" ]
split => [ "custom_data", ";" ]
}
grok {
match => [ "source_context", "(-|%{WORD:MDB}),%{SPACE}(-|%{WORD:Mailbox}),%{SPACE}%{INTEGER:Event},%{SPACE}%{WORD:MessageClass},%{SPACE}%{TIMESTAMP_ISO8601:CreationTime},%{SPACE}%{WORD:ClientType}" ]
}
date {
match => [ "timestamp", "ISO8601" ]
timezone => "Europe/London"
remove_field => [ "timestamp" ]
}
if "_grokparsefailure" in [tags] {
drop { }
}
}
0x00000000000fc4 entries=2 default_proc=nil>, @structured_lookup_mutex=#