Parsing Windows Event Logs with nested xml

[jmohar]

Dear community.
After spending more than a week with logstash and winlogbeat to parse Windows_Event_Logs..but have no clue how to parse the nested xml in the security windows eventlogs (vistaeventlogs).

Using Logstash 6.6.1

I learned that Winlogbeat is not able to parse the nested xml in this event_log . So we ship the event_log to logstash, which contain this message_field:

message" => "The Federation Service issued a valid token. See XML for details. \n\nActivity ID: 04ca7383-37f9-4d06-bb5f-0080010000f4 \n\nAdditional Data \nXML: <?xml version=\"1.0\" encoding=\"utf-16\"?>\n<AuditBase xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:type=\"AppTokenAudit\">\n <AuditType>AppToken</AuditType>\n <AuditResult>Success</AuditResult>\n <FailureType>None</FailureType>\n <ErrorCode>N/A</ErrorCode>\n <ContextComponents>\n <Component xsi:type=\"ResourceAuditComponent\">\n <RelyingParty>https://gym-subs.ro.r4.madm.net/saml/module.php/saml/sp/metadata.php/gym-subs-sp</RelyingParty>\n <ClaimsProvider>AD AUTHORITY</ClaimsProvider>\n <UserId>testuser</UserId>\n </Component>\n <Component xsi:type=\"AuthNAuditComponent\">\n <PrimaryAuth>http://schemas.microsoft.com/ws/2008/06/identity/authenticationmethod/windows</PrimaryAuth>\n <DeviceAuth>false</DeviceAuth>\n <DeviceId>N/A</DeviceId>\n <MfaPerformed>false</MfaPerformed>\n <MfaMethod>N/A</MfaMethod>\n <TokenBindingProvidedId>false</TokenBindingProvidedId>\n <TokenBindingReferredId>false</TokenBindingReferredId>\n <SsoBindingValidationLevel>TokenUnbound</SsoBindingValidationLevel>\n </Component>\n <Component xsi:type=\"ProtocolAuditComponent\">\n <OAuthClientId>N/A</OAuthClientId>\n <OAuthGrant>N/A</OAuthGrant>\n </Component>\n <Component xsi:type=\"RequestAuditComponent\">\n <Server>http://adfs4int.fds.metro.info/adfs/services/trust</Server>\n <AuthProtocol>SAMLP</AuthProtocol>\n <NetworkLocation>Intranet</NetworkLocation>\n <IpAddress>10.16.234.42</IpAddress>\n <ForwardedIpAddress />\n <ProxyIpAddress>N/A</ProxyIpAddress>\n <NetworkIpAddress>N/A</NetworkIpAddress>\n <ProxyServer>N/A</ProxyServer>\n <UserAgentString>Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0</UserAgentString>\n <Endpoint>/adfs/ls/</Endpoint>\n </Component>\n </ContextComponents>\n</AuditBase>"

Using the xml filter this message is parsed to "xml.ContextComponents.Component"
Unfortunately i was not able to "xpath" necessary attributes from "messages"...also "grok" this message is more or less frustrating

Searching the Community i found a hint to split the message field after running this through the xml filter...this seems to work...but unfortunately creates 3 additional events plus the original event.
Is there a way to aggregate this 4 events back to one , again ?

Out logstash.config:

input {
  beats {
    port => 5044
	}
}
filter {
  xml {
      source => "message"
      store_xml => "true"
      target => "xml"
     force_array => false
    } 
    split { field => "[xml][ContextComponents][Component]" }
  }

output {
  stdout { codec => rubydebug }
}

Do you have any hints, what is the right way to parse this "message" field with nested xml ?

[Badger]

@jmohar

Searching the Community i found a hint to split the message field after running this through the xml filter...this seems to work...but unfortunately creates 3 additional events plus the original event.
Is there a way to aggregate this 4 events back to one , again ?

The entire point of a split filter is to split one event into more than one. If you want to keep one event as one event then do not use a split filter.

The following xpath expressions will pull items out of the Component entries.

        xpath => {
            "/AuditBase/ContextComponents/Component/ClaimsProvider/text()" => "ClaimsProvider"
            "/AuditBase/ContextComponents/Component/PrimaryAuth/text()" => "PrimaryAuth"
            "/AuditBase/ContextComponents/Component/IpAddress/text()" => "IpAddress"
        }

If that does not help then please explain what you are trying to do.

[jmohar]

@jmohar

UserId

Thank you for the quick reply! I already "played around" with xpath...but regardless what path setting i insert...the xpath field are always empty:

I tested various xpath settings...the fields are always empty in stndout and kibana

filter {
  xml {
      source => "message"
      store_xml => "false"
     force_array => false
        xpath => {
	    "/AuditBase/AuditType/text()" => "AuditType"
   	    "/AuditBase/AuditResult/text()" => "AuditResult"
            "/AuditBase/ContextComponents/Component/RelyingParty" => "RelyingParty"
            "/AuditBase/ContextComponents/Component/ClaimsProvider/text()" => "ClaimsProvider"
            "/AuditBase/ContextComponents/Component/UserId/text()" => "UserId"
        }
    }

But the fields are always empty:

STND shows this:
Mar 18 09:06:57 SSP30MYSRV01394 logstash[16395]: "RelyingParty" => ,
Mar 18 09:06:57 SSP30MYSRV01394 logstash[16395]: "ClaimsProvider" => ,

Do you have any further hints for me ?

[Badger]

Are you removing the prefix before you parse the XML?

   mutate { gsub => [ "message", "(?m).*XML: ", "" ] }

I was surprised to find that the xml filter skips the prefix, but none of the xpath entries work. Once you remove the prefix xpath starts working.

[jmohar]

Beside the "message" field... also "event_data.param1" and "event_data.param2" was shipped by winlogbeat. "event_data.param2" contain pure xml data. "message" is somekind of "whatever"...strings with embedded xml. Nothing logstash can work with out of the box. Parsing the source "event_data.param2" does the trick and xpath was able to find the attributes. Thank you for the hint !

This does the trick for us:

filter {
  mutate {
    add_field => {"Activity_ID" => "%{[event_data][param1]}"}
  }
  xml {
      source => "[event_data][param2]"
      store_xml => "false"
      force_array => false
        xpath => {
          "/AuditBase/AuditType/text()" => "AuditType"
          "/AuditBase/AuditResult/text()" => "AuditResult"
          "/AuditBase/FailureType/text()" => "FailureType"
	      "/AuditBase/ErrorCode/text()" => "ErrorCode"
	      "/AuditBase/ContextComponents/Component/RelyingParty/text()" => "RelyingParty"
          "/AuditBase/ContextComponents/Component/ClaimsProvider/text()" => "ClaimsProvider"
          "/AuditBase/ContextComponents/Component/UserId/text()" => "UserId"
	      "/AuditBase/ContextComponents/Component/OAuthClientId/text()" => "OAuthClientId"
	      "/AuditBase/ContextComponents/Component/OAuthGrant/text()" => "OAuthGrant"
	      "/AuditBase/ContextComponents/Component/Server/text()" => "Server"
	      "/AuditBase/ContextComponents/Component/AuthProtocol/text()" => "AuthProtocol"
        }
    }