We are polling a security web service to get the scanning and detection logs.
This is what I get when I poll the service with logstash:
{
"current_link" => "https://xxx/v1/siem/security_events?service=exchange&event=securityrisk",
"@timestamp" => 2019-09-05T00:17:30.019Z,
"last_log_item_generation_time" => "2019-09-05T00:15:59.000Z",
"traceId" => "aa157e14-b5c4-44ed-a12f-230f42ce4e2d",
"@version" => "1",
"next_link" => "",
"security_events" => [
[0] {
"message" => {
"mail_message_subject" => "Zakończenie konkursu - Wygraj bilety do kina",
"security_risk_name" => "Other spam",
"detected_by" => "Antispam engine",
"affected_user" => "xxx",
"triggered_security_filter" => "Advanced Spam Protection",
"mail_message_id" => "<xxx>",
"action_result" => "success",
"scan_type" => "Real-time scan",
"risk_level" => "",
"location" => "xxx",
"triggered_policy_name" => "Default Exchange Policy ATP",
"detection_time" => "2019-09-05T00:15:59.000Z",
"mail_message_file_name" => "",
"action" => "Move to Junk Email folder",
"mail_message_recipient" => [
[0] "\"xxx\"<xxx>"
],
"mail_message_submit_time" => "2019-09-05T00:15:54.000Z",
"mail_message_delivery_time" => "2019-09-05T00:01:31.000Z",
"mail_message_sender" => "\"xxx\"xxx>"
},
"service" => "Exchange Online",
"event" => "security_risk_scan",
"log_item_id" => "TyDH_mwBsV0XoBdHJoOn"
}
[1] {
"message" => {
"mail_message_subject" => "xxx",
"security_risk_name" => "Other spam",
"detected_by" => "Antispam engine",
"affected_user" => "xxx",
"triggered_security_filter" => "Advanced Spam Protection",
"mail_message_id" => "<xxx>",
"action_result" => "success",
"scan_type" => "Real-time scan",
"risk_level" => "",
"location" => "xxx",
"triggered_policy_name" => "Default Exchange Policy ATP",
"detection_time" => "2019-09-05T00:18:45.000Z",
"mail_message_file_name" => "",
"action" => "Move to Junk Email folder",
"mail_message_recipient" => [
[0] "\"xxx\"<xxx>"
],
"mail_message_submit_time" => "2019-09-05T00:18:38.000Z",
"mail_message_delivery_time" => "2019-09-05T00:14:06.000Z",
"mail_message_sender" => "\"xxx\"<xxx>"
},
"service" => "Exchange Online",
"event" => "security_risk_scan",
"log_item_id" => "_YzJ_mwBhwpaNQMzrmMf"
}
]}
I have quite a bit expirience with logstash and plain logs, but I have no idea how to parse this JSON? construct.
One of the problems is that we are parsing the web service so the security_events filed can contain variuos entries.
What I like to accomplish is that I get one elasticsearch entry for every entry in the security_events field.