Need some help with Logstash and the XML filter plugin

riahc3 · February 23, 2021, 4:29pm

Hello

Im trying to parse a XML that Im getting from a Windows Event Viewer.

Ill try to explain as best I can and any questions or doubt can be further asked.

I have this in a field that is winlog.event_data.param2 (values have been edited for privacy reasons) ::

<?xml version="1.0" encoding="utf-16"?>
<AuditBase xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="FreshCredentialAudit">
  <AuditType>FreshCredentials</AuditType>
  <AuditResult>Failure</AuditResult>
  <FailureType>CredentialValidationError</FailureType>
  <ErrorCode>N/A</ErrorCode>
  <ContextComponents>
    <Component xsi:type="ResourceAuditComponent">
      <RelyingParty>N/A</RelyingParty>
      <ClaimsProvider>N/A</ClaimsProvider>
      <UserId>auser@adomain.com</UserId>
    </Component>
    <Component xsi:type="AuthNAuditComponent">
      <PrimaryAuth>N/A</PrimaryAuth>
      <DeviceAuth>false</DeviceAuth>
      <DeviceId>N/A</DeviceId>
      <MfaPerformed>false</MfaPerformed>
      <MfaMethod>N/A</MfaMethod>
      <TokenBindingProvidedId>false</TokenBindingProvidedId>
      <TokenBindingReferredId>false</TokenBindingReferredId>
      <SsoBindingValidationLevel>NotSet</SsoBindingValidationLevel>
    </Component>
    <Component xsi:type="ProtocolAuditComponent">
      <OAuthClientId>N/A</OAuthClientId>
      <OAuthGrant>N/A</OAuthGrant>
    </Component>
    <Component xsi:type="RequestAuditComponent">
      <Server>http://fs.adomain.com/adfs/services/trust</Server>
      <AuthProtocol>N/A</AuthProtocol>
      <NetworkLocation>Extranet</NetworkLocation>
      <IpAddress>123.123.123.123,231.231.231.231</IpAddress>
      <ForwardedIpAddress>123.123.123.123,231.231.231.231</ForwardedIpAddress>
      <ProxyIpAddress>N/A</ProxyIpAddress>
      <NetworkIpAddress>N/A</NetworkIpAddress>
      <ProxyServer>ASERVERTHATISADC</ProxyServer>
      <UserAgentString>Microsoft Office/16.0 (Windows NT 10.0; Microsoft Outlook 16.0.13628; Pro)</UserAgentString>
      <Endpoint>/adfs/services/trust/2005/usernamemixed</Endpoint>
    </Component>
  </ContextComponents>
</AuditBase>

My original Logstash config is:

input {
  beats {
    port => 5051
        tags => [ "winlogbeat" ]
  }
}

filter

{

  if "winlogbeat" not in [tags]

 {

        drop { }

   }

 }



filter

{

  if [winlog][channel] != "Security"

 {

        drop { }

   }

 }



filter

{

  if [event][code] != 1200 and [event][code] != 1202 and [event][code] != 1203

 {

        drop { }

   }

 }





filter {

mutate {
    copy => { "[host][hostname]" => "[@metadata][hostlower]" }
}
mutate {
    lowercase => [ "[@metadata][hostlower]" ]
}
}


output
{

    elasticsearch {

hosts => ["localhost"]
user => ["elastic"]
password  => ["apassword"]

 index => "aindex-%{+yyyy.MM.dd}"

        }

}

My idea/thought process (but doesnt work) is:

input {
  beats {
    port => 5051
        tags => [ "winlogbeat" ]
  }
}

filter

{

  if "winlogbeat" not in [tags]

 {

        drop { }

   }

 }



filter

{

  if [winlog][channel] != "Security"

 {

        drop { }

   }

 }



filter

{

  if [event][code] != 1200 and [event][code] != 1202 and [event][code] != 1203

 {

        drop { }

   }

 }



filter {
        xml{
                store_xml => false
                source => [winlog][event_data][param2]
                xpath => ["/AuditBase/AuditType/text()", "AuditType"]
        }
}



filter {

mutate {
    copy => { "[host][hostname]" => "[@metadata][hostlower]" }
}
mutate {
    lowercase => [ "[@metadata][hostlower]" ]
}
}


output
{

    elasticsearch {

hosts => ["localhost"]
user => ["elastic"]
password  => ["apassword"]

 index => "aindex-%{+yyyy.MM.dd}"

        }

}

Can someone lend me a hand and help me understand and how to do this?

Thank you very much

riahc3 · February 24, 2021, 5:22pm

No ideas on how to tackle this or where to start looking?

Thank you

Badger · February 24, 2021, 6:15pm

What do you mean by that? With that data and that xml filter I get

 "AuditType" => [
    [0] "FreshCredentials"
],

So it works for me.

riahc3 · February 25, 2021, 2:13pm

I get this error message:

[2021-02-25T13:49:41,101][ERROR][logstash.agent           ] Failed to execute action {:id=>:adfs, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Expected one of [ \\t\\r\\n], \"#\", \"}\" at line 59, column 35 (byte 447) after filter {\n        xml{\n                store_xml => false\n                source => [winlog]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/reload.rb:53:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:365:in `block in converge_state'"]}
[2021-02-25T13:49:44,195][ERROR][logstash.agent           ] Failed to execute action {:id=>:adfs, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Expected one of [ \\t\\r\\n], \"#\", \"}\" at line 59, column 35 (byte 447) after filter {\n        xml{\n                store_xml => false\n                source => [winlog]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/reload.rb:53:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:365:in `block in converge_state'"]}

But are you saying it works for you? Or it doesnt work as expected?

Badger · February 25, 2021, 4:22pm

You need double quotes around "[winlog][event_data][param2]"

riahc3 · February 25, 2021, 4:40pm

Thanks Thats seems to have worked fine; Im gonna keep on breaking down the rest of the fields.

I do see that the field has been introduced as a unknown type: I would like to give it a type such as string, number, etc. How can I do this?

Badger · February 25, 2021, 4:46pm

The only way I can think of for you to see an unknown type is in kibana, if you have not done an index pattern refresh. Otherwise I think everything in logstash will be a string and in elasticsearch dynamic field mapping will have typed things.

riahc3 · February 25, 2021, 5:05pm

Yup, thats problably it. I think I had to do that once.

Im having another issue (even though its dumb)

This works:

filter {
        xml {
                store_xml => false
                source => "[winlog][event_data][param2]"
                xpath => ["/AuditBase/AuditType/text()", "AuditType"]
				xpath => ["/AuditBase/AuditResult/text()", "AuditResult"]
				xpath => ["/AuditBase/FailureType/text()", "FailureType"]
				xpath => ["/AuditBase/ErrorCode/text()", "ErrorCode"]
        }
}

This does NOT work

filter {
        xml {
                store_xml => false
                source => "[winlog][event_data][param2]"
                xpath => ["/AuditBase/AuditType/text()", "AuditType"]
				xpath => ["/AuditBase/AuditResult/text()", "AuditResult"]
				xpath => ["/AuditBase/FailureType/text()", "FailureType"]
				xpath => ["/AuditBase/ErrorCode/text()", "ErrorCode"]
				xpath => ["/AuditBase/ContextComponents/Component/RelyingParty/text()", "RelyingParty"]
        }
}

As you can see that last field is inside another XML field; Maybe Im not getting it right. Could you clarify it for me?

Thank you

Badger · February 25, 2021, 5:54pm

riahc3:

                xpath => ["/AuditBase/AuditType/text()", "AuditType"]
				xpath => ["/AuditBase/AuditResult/text()", "AuditResult"]
				xpath => ["/AuditBase/FailureType/text()", "FailureType"]
				xpath => ["/AuditBase/ErrorCode/text()", "ErrorCode"]
				xpath => ["/AuditBase/ContextComponents/Component/RelyingParty/text()", "RelyingParty"]

If you supply an option multiple times logstash will combine them. Usually it does that in exactly the way you would expect. From time to time I have found cases where it does not. So I only use a single instance of any given option

        xpath => {
            "/AuditBase/AuditType/text()" => "AuditType"
            "/AuditBase/AuditResult/text()" => "AuditResult"
            "/AuditBase/FailureType/text()" => "FailureType"
            "/AuditBase/ErrorCode/text()" => "ErrorCode"
            "/AuditBase/ContextComponents/Component/RelyingParty/text()" => "RelyingParty"
        }

That said, even with 5 options I get

 "FailureType" => [
    [0] "CredentialValidationError"
],
 "AuditResult" => [
    [0] "Failure"
],
"RelyingParty" => [
    [0] "N/A"
],
   "AuditType" => [
    [0] "FreshCredentials"
],
   "ErrorCode" => [
    [0] "N/A"
],

So I am not sure what you might be doing wrong.

riahc3 · February 26, 2021, 8:45am

Combining as one DOES seem to work..... (a bunch inside ONE xpath instead of multiple xpath)

riahc3 · February 26, 2021, 8:47am

Hmmm.........Ever since I implemented this though, Kibana is acting funky: Throwing me random 500 errors. And other errors:

{"statusCode":429,"error":"Too Many Requests","message":"[parent] Data too large, data for [<http_request>] would be [1033145744/985.2mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1033145744/985.2mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=18361954/17.5mb, in_flight_requests=0/0b, model_inference=0/0b, accounting=75981320/72.4mb]: circuit_breaking_exception"}

system · March 26, 2021, 8:47am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
WinlogBeat -> LogStash XML filter Logstash	6	1537	March 30, 2018
Help using Logstash XML Filter to Parse Offline Windows Event Logs Logstash	5	3082	June 23, 2018
Parsing Issue for XML File Logstash	2	530	November 24, 2017
Logstash XML parsing issues - trying to send to Graylog, part 2 Logstash	5	986	August 28, 2019
Not properly formated XML Logstash	1	252	December 27, 2020

Need some help with Logstash and the XML filter plugin

Related Topics