Filebeat multiline - how to tell filebeat when a message ends while parsing Microsoft Defender ATP logs

Elastic Stack Beats
1

Hello, I'm trying to integrate Microsoft Defender ATP with Graylog, and I'm pulling in events using the Microsoft API. The output is spread across multiple lines, so I'm using the multiline function within filebeat to pull in the entire event, and not just each line as a separate message.

The problem I'm having, is when there is more than 1 event in the log - Filebeat reads them all as a big chunk. Is there a way to tell filebeat when to end processing a multiline and start a new one?

I tried a multiline_flush_pattern, but that just results in nothing making it into graylog:
multiline.flush_pattern: '},'

I’m using the following in the filebeat config:

multiline.pattern: '\"id\"'
multiline.negate: true
multiline.match: after

Here's a sample log:

[
    {
        "id":  "12345",
        "providerAlertId":  "1234567",
        "incidentId":  "29",
        "status":  "resolved",
        "severity":  "medium",
        "classification":  null,
        "determination":  null,
        "serviceSource":  "microsoftDefenderForEndpoint",
        "detectionSource":  "antivirus",
        "detectorId":  "abcdefg12345",
        "tenantId":  "abcdefg12345",
        "title":  "Bloodhound post-exploitation tool",
        "description":  "Bloodhound, a post-exploitation open-source reconnaissance tool, has been detected on this device. Bloodhound has been used in a wide range of documented attacks, including attacks involving state-sponsored groups and groups associated with ransomware campaigns. An attacker might be attempting to collect information about users, user sessions, groups, accounts, domain controller properties and permissions. Detections of Bloodhound tools and activity should be thoroughly investigated.",
        "recommendedActions":  "A. Validate the alert.\n1. Contact the user who ran the tool to verify whether the activity was legitimate and inspect the endpoints for suspicious behavior.\n2. Locate unfamiliar processes in the process tree. Check files for prevalence, their locations, and digital signatures.\n3. Submit relevant files for deep analysis and review file behaviors.\n4. Identify unusual system activities with system owners.\n\nB. Scope the incident. Find related devices, network addresses, and files in the incident graph.\n\nC. Contain and mitigate the breach. Stop suspicious processes, isolate affected devices, decommission compromised accounts, or reset passwords, block IP addresses and URLs, and install security updates.\n\nD. Contact your incident response team, or contact Microsoft support for investigation and remediation services.",
        "category":  "SuspiciousActivity",
        "assignedTo":  "Automation",
        "alertWebUrl":  "https://security.microsoft.com/alerts/blahblahblahblah",
        "incidentWebUrl":  "https://security.microsoft.com/incidents/blahblahblah",
        "actorDisplayName":  null,
        "threatDisplayName":  "HackTool:MSIL/SharpHound!MSR",
        "threatFamilyName":  "SharpHound",
        "mitreTechniques":  [
                                "T1087"
                            ],
        "createdDateTime":  "2022-12-21T18:40:21.7967173Z",
        "lastUpdateDateTime":  "2022-12-21T18:52:56.91Z",
        "resolvedDateTime":  "2022-12-21T18:52:56.7897235Z",
        "firstActivityDateTime":  "2022-12-21T18:38:49.4766284Z",
        "lastActivityDateTime":  "2022-12-21T18:48:11.3281966Z",

    },
    
	{
        "id":  "1234567",
        "providerAlertId":  "1234567",
        "incidentId":  "29",
        "status":  "resolved",
        "severity":  "medium",
        "classification":  null,
        "determination":  null,
        "serviceSource":  "microsoftDefenderForEndpoint",
        "detectionSource":  "microsoftDefenderForEndpoint",
        "detectorId":  "1a2b3c4d",
        "tenantId":  "1a2b3c4d5e",
        "title":  "\u0027SharpHound\u0027 malware was detected during lateral movement",
        "description":  "\u0027DefenderObservation.ThreatFamily\u0027 malware was detected on this device. An attacker might be attempting to move laterally to this device from another device on the network.",
        "recommendedActions":  "A. Validate the alert.\n1. Inspect the file, process, commandline or amsi buffer that was detected.\n2. Check for other suspicious activities in the machine timeline.​\n3. Locate unfamiliar processes in the process tree. Check files for prevalence, their locations, and digital signatures.​\n4. Submit relevant files for deep analysis and review file behaviors. ​\n5. Identify unusual system activity with system owners.​\n​\nB. Scope the incident. Find related machines, network addresses, and files in the incident graph.​\n​\nC. Contain and mitigate the breach. Stop suspicious processes, isolate affected machines, decommission compromised accounts or reset passwords, block IP addresses and URLs, and install security updates.​\n​\nD. Contact your incident response team, or contact Microsoft support for investigation and remediation services.",
        "category":  "LateralMovement",
        "assignedTo":  "Automation",
        "alertWebUrl":  "https://security.microsoft.com/alerts/12345_12345",
        "incidentWebUrl":  "https://security.microsoft.com/incidents/29?tid=1ab3ddddd",
        "actorDisplayName":  null,
        "threatDisplayName":  null,
        "threatFamilyName":  null,
        "mitreTechniques":  [
                                "T1021.001",
                                "T1078"
                            ],
        "createdDateTime":  "2022-12-21T18:40:49.4268815Z",
        "lastUpdateDateTime":  "2022-12-21T18:52:56.91Z",
        "resolvedDateTime":  "2022-12-21T18:52:56.7897235Z",
        "firstActivityDateTime":  "2022-12-21T18:06:37.0236159Z",
        "lastActivityDateTime":  "2022-12-21T18:47:23.7266277Z",
        
    },
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.