Need some help with Logstash and the XML filter plugin

Hello

Im trying to parse a XML that Im getting from a Windows Event Viewer.

Ill try to explain as best I can and any questions or doubt can be further asked.

I have this in a field that is winlog.event_data.param2 (values have been edited for privacy reasons) ::

<?xml version="1.0" encoding="utf-16"?>
<AuditBase xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="FreshCredentialAudit">
  <AuditType>FreshCredentials</AuditType>
  <AuditResult>Failure</AuditResult>
  <FailureType>CredentialValidationError</FailureType>
  <ErrorCode>N/A</ErrorCode>
  <ContextComponents>
    <Component xsi:type="ResourceAuditComponent">
      <RelyingParty>N/A</RelyingParty>
      <ClaimsProvider>N/A</ClaimsProvider>
      <UserId>auser@adomain.com</UserId>
    </Component>
    <Component xsi:type="AuthNAuditComponent">
      <PrimaryAuth>N/A</PrimaryAuth>
      <DeviceAuth>false</DeviceAuth>
      <DeviceId>N/A</DeviceId>
      <MfaPerformed>false</MfaPerformed>
      <MfaMethod>N/A</MfaMethod>
      <TokenBindingProvidedId>false</TokenBindingProvidedId>
      <TokenBindingReferredId>false</TokenBindingReferredId>
      <SsoBindingValidationLevel>NotSet</SsoBindingValidationLevel>
    </Component>
    <Component xsi:type="ProtocolAuditComponent">
      <OAuthClientId>N/A</OAuthClientId>
      <OAuthGrant>N/A</OAuthGrant>
    </Component>
    <Component xsi:type="RequestAuditComponent">
      <Server>http://fs.adomain.com/adfs/services/trust</Server>
      <AuthProtocol>N/A</AuthProtocol>
      <NetworkLocation>Extranet</NetworkLocation>
      <IpAddress>123.123.123.123,231.231.231.231</IpAddress>
      <ForwardedIpAddress>123.123.123.123,231.231.231.231</ForwardedIpAddress>
      <ProxyIpAddress>N/A</ProxyIpAddress>
      <NetworkIpAddress>N/A</NetworkIpAddress>
      <ProxyServer>ASERVERTHATISADC</ProxyServer>
      <UserAgentString>Microsoft Office/16.0 (Windows NT 10.0; Microsoft Outlook 16.0.13628; Pro)</UserAgentString>
      <Endpoint>/adfs/services/trust/2005/usernamemixed</Endpoint>
    </Component>
  </ContextComponents>
</AuditBase>

My original Logstash config is:

input {
  beats {
    port => 5051
        tags => [ "winlogbeat" ]
  }
}

filter

{

  if "winlogbeat" not in [tags]

 {

        drop { }

   }

 }



filter

{

  if [winlog][channel] != "Security"

 {

        drop { }

   }

 }



filter

{

  if [event][code] != 1200 and [event][code] != 1202 and [event][code] != 1203

 {

        drop { }

   }

 }





filter {

mutate {
    copy => { "[host][hostname]" => "[@metadata][hostlower]" }
}
mutate {
    lowercase => [ "[@metadata][hostlower]" ]
}
}


output
{

    elasticsearch {

hosts => ["localhost"]
user => ["elastic"]
password  => ["apassword"]

 index => "aindex-%{+yyyy.MM.dd}"

        }

}

My idea/thought process (but doesnt work) is:

input {
  beats {
    port => 5051
        tags => [ "winlogbeat" ]
  }
}

filter

{

  if "winlogbeat" not in [tags]

 {

        drop { }

   }

 }



filter

{

  if [winlog][channel] != "Security"

 {

        drop { }

   }

 }



filter

{

  if [event][code] != 1200 and [event][code] != 1202 and [event][code] != 1203

 {

        drop { }

   }

 }



filter {
        xml{
                store_xml => false
                source => [winlog][event_data][param2]
                xpath => ["/AuditBase/AuditType/text()", "AuditType"]
        }
}



filter {

mutate {
    copy => { "[host][hostname]" => "[@metadata][hostlower]" }
}
mutate {
    lowercase => [ "[@metadata][hostlower]" ]
}
}


output
{

    elasticsearch {

hosts => ["localhost"]
user => ["elastic"]
password  => ["apassword"]

 index => "aindex-%{+yyyy.MM.dd}"

        }

}

Can someone lend me a hand and help me understand and how to do this?

Thank you very much

No ideas on how to tackle this or where to start looking?

Thank you

What do you mean by that? With that data and that xml filter I get

 "AuditType" => [
    [0] "FreshCredentials"
],

So it works for me.

I get this error message:

[2021-02-25T13:49:41,101][ERROR][logstash.agent           ] Failed to execute action {:id=>:adfs, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Expected one of [ \\t\\r\\n], \"#\", \"}\" at line 59, column 35 (byte 447) after filter {\n        xml{\n                store_xml => false\n                source => [winlog]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/reload.rb:53:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:365:in `block in converge_state'"]}
[2021-02-25T13:49:44,195][ERROR][logstash.agent           ] Failed to execute action {:id=>:adfs, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Expected one of [ \\t\\r\\n], \"#\", \"}\" at line 59, column 35 (byte 447) after filter {\n        xml{\n                store_xml => false\n                source => [winlog]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/reload.rb:53:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:365:in `block in converge_state'"]}

But are you saying it works for you? Or it doesnt work as expected?

You need double quotes around "[winlog][event_data][param2]"

Thanks :slight_smile: Thats seems to have worked fine; Im gonna keep on breaking down the rest of the fields.

I do see that the field has been introduced as a unknown type: I would like to give it a type such as string, number, etc. How can I do this?

The only way I can think of for you to see an unknown type is in kibana, if you have not done an index pattern refresh. Otherwise I think everything in logstash will be a string and in elasticsearch dynamic field mapping will have typed things.

Yup, thats problably it. I think I had to do that once.

Im having another issue (even though its dumb)

This works:

filter {
        xml {
                store_xml => false
                source => "[winlog][event_data][param2]"
                xpath => ["/AuditBase/AuditType/text()", "AuditType"]
				xpath => ["/AuditBase/AuditResult/text()", "AuditResult"]
				xpath => ["/AuditBase/FailureType/text()", "FailureType"]
				xpath => ["/AuditBase/ErrorCode/text()", "ErrorCode"]
        }
}

This does NOT work

filter {
        xml {
                store_xml => false
                source => "[winlog][event_data][param2]"
                xpath => ["/AuditBase/AuditType/text()", "AuditType"]
				xpath => ["/AuditBase/AuditResult/text()", "AuditResult"]
				xpath => ["/AuditBase/FailureType/text()", "FailureType"]
				xpath => ["/AuditBase/ErrorCode/text()", "ErrorCode"]
				xpath => ["/AuditBase/ContextComponents/Component/RelyingParty/text()", "RelyingParty"]
        }
}

As you can see that last field is inside another XML field; Maybe Im not getting it right. Could you clarify it for me?

Thank you

If you supply an option multiple times logstash will combine them. Usually it does that in exactly the way you would expect. From time to time I have found cases where it does not. So I only use a single instance of any given option

        xpath => {
            "/AuditBase/AuditType/text()" => "AuditType"
            "/AuditBase/AuditResult/text()" => "AuditResult"
            "/AuditBase/FailureType/text()" => "FailureType"
            "/AuditBase/ErrorCode/text()" => "ErrorCode"
            "/AuditBase/ContextComponents/Component/RelyingParty/text()" => "RelyingParty"
        }

That said, even with 5 options I get

 "FailureType" => [
    [0] "CredentialValidationError"
],
 "AuditResult" => [
    [0] "Failure"
],
"RelyingParty" => [
    [0] "N/A"
],
   "AuditType" => [
    [0] "FreshCredentials"
],
   "ErrorCode" => [
    [0] "N/A"
],

So I am not sure what you might be doing wrong.

Combining as one DOES seem to work..... (a bunch inside ONE xpath instead of multiple xpath)

Hmmm.........Ever since I implemented this though, Kibana is acting funky: Throwing me random 500 errors. And other errors:

{"statusCode":429,"error":"Too Many Requests","message":"[parent] Data too large, data for [<http_request>] would be [1033145744/985.2mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1033145744/985.2mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=18361954/17.5mb, in_flight_requests=0/0b, model_inference=0/0b, accounting=75981320/72.4mb]: circuit_breaking_exception"}