# Need some help with Logstash and the XML filter plugin

**URL:** <https://discuss.elastic.co/t/need-some-help-with-logstash-and-the-xml-filter-plugin/265227>\
**Category:** Logstash\
**Created:** [February 23, 2021, 4:29pm UTC](https://discuss.elastic.co/t/need-some-help-with-logstash-and-the-xml-filter-plugin/265227 "2021-02-23T16:29:57Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![riahc3](https://avatars.discourse-cdn.com/v4/letter/r/d6d6ee/32.png) [@riahc3](https://discuss.elastic.co/u/riahc3)\
**Post date:** [February 23, 2021, 4:29pm UTC](https://discuss.elastic.co/t/need-some-help-with-logstash-and-the-xml-filter-plugin/265227/1 "2021-02-23T16:29:57Z")

</div>

Hello

Im trying to parse a XML that Im getting from a Windows Event Viewer.

Ill try to explain as best I can and any questions or doubt can be further asked.

I have this in a field that is winlog.event\_data.param2 (values have been edited for privacy reasons) ::

```
<?xml version="1.0" encoding="utf-16"?>
<AuditBase xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="FreshCredentialAudit">
  <AuditType>FreshCredentials</AuditType>
  <AuditResult>Failure</AuditResult>
  <FailureType>CredentialValidationError</FailureType>
  <ErrorCode>N/A</ErrorCode>
  <ContextComponents>
    <Component xsi:type="ResourceAuditComponent">
      <RelyingParty>N/A</RelyingParty>
      <ClaimsProvider>N/A</ClaimsProvider>
      <UserId>auser@adomain.com</UserId>
    </Component>
    <Component xsi:type="AuthNAuditComponent">
      <PrimaryAuth>N/A</PrimaryAuth>
      <DeviceAuth>false</DeviceAuth>
      <DeviceId>N/A</DeviceId>
      <MfaPerformed>false</MfaPerformed>
      <MfaMethod>N/A</MfaMethod>
      <TokenBindingProvidedId>false</TokenBindingProvidedId>
      <TokenBindingReferredId>false</TokenBindingReferredId>
      <SsoBindingValidationLevel>NotSet</SsoBindingValidationLevel>
    </Component>
    <Component xsi:type="ProtocolAuditComponent">
      <OAuthClientId>N/A</OAuthClientId>
      <OAuthGrant>N/A</OAuthGrant>
    </Component>
    <Component xsi:type="RequestAuditComponent">
      <Server>http://fs.adomain.com/adfs/services/trust</Server>
      <AuthProtocol>N/A</AuthProtocol>
      <NetworkLocation>Extranet</NetworkLocation>
      <IpAddress>123.123.123.123,231.231.231.231</IpAddress>
      <ForwardedIpAddress>123.123.123.123,231.231.231.231</ForwardedIpAddress>
      <ProxyIpAddress>N/A</ProxyIpAddress>
      <NetworkIpAddress>N/A</NetworkIpAddress>
      <ProxyServer>ASERVERTHATISADC</ProxyServer>
      <UserAgentString>Microsoft Office/16.0 (Windows NT 10.0; Microsoft Outlook 16.0.13628; Pro)</UserAgentString>
      <Endpoint>/adfs/services/trust/2005/usernamemixed</Endpoint>
    </Component>
  </ContextComponents>
</AuditBase>

```

My original Logstash config is:

```
input {
  beats {
    port => 5051
        tags => ["winlogbeat"]
  }
}

filter

{

  if "winlogbeat" not in [tags]

 {

        drop { }

   }

 }

filter

{

  if [winlog][channel] != "Security"

 {

        drop { }

   }

 }

filter

{

  if [event][code] != 1200 and [event][code] != 1202 and [event][code] != 1203

 {

        drop { }

   }

 }

filter {

mutate {
    copy => { "[host][hostname]" => "[@metadata][hostlower]" }
}
mutate {
    lowercase => ["[@metadata][hostlower]" ]
}
}

output
{

    elasticsearch {

hosts => ["localhost"]
user => ["elastic"]
password => ["apassword"]

 index => "aindex-%{+yyyy.MM.dd}"

        }

}

```

My idea/thought process (but doesnt work) is:

```
input {
  beats {
    port => 5051
        tags => ["winlogbeat"]
  }
}

filter

{

  if "winlogbeat" not in [tags]

 {

        drop { }

   }

 }

filter

{

  if [winlog][channel] != "Security"

 {

        drop { }

   }

 }

filter

{

  if [event][code] != 1200 and [event][code] != 1202 and [event][code] != 1203

 {

        drop { }

   }

 }

filter {
        xml{
                store_xml => false
                source => [winlog][event_data][param2]
                xpath => ["/AuditBase/AuditType/text()", "AuditType"]
        }
}

filter {

mutate {
    copy => { "[host][hostname]" => "[@metadata][hostlower]" }
}
mutate {
    lowercase => ["[@metadata][hostlower]" ]
}
}

output
{

    elasticsearch {

hosts => ["localhost"]
user => ["elastic"]
password => ["apassword"]

 index => "aindex-%{+yyyy.MM.dd}"

        }

}

```

Can someone lend me a hand and help me understand and how to do this?

Thank you very much

---

<div class="post-metadata">

**Author:** ![riahc3](https://avatars.discourse-cdn.com/v4/letter/r/d6d6ee/32.png) [@riahc3](https://discuss.elastic.co/u/riahc3)\
**Post date:** [February 24, 2021, 5:22pm UTC](https://discuss.elastic.co/t/need-some-help-with-logstash-and-the-xml-filter-plugin/265227/2 "2021-02-24T17:22:57Z")

</div>

No ideas on how to tackle this or where to start looking?

Thank you

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 24, 2021, 6:15pm UTC](https://discuss.elastic.co/t/need-some-help-with-logstash-and-the-xml-filter-plugin/265227/3 "2021-02-24T18:15:59Z")

</div>

> [@riahc3](#):
>
> but doesnt work

What do you mean by that? With that data and that xml filter I get

```
 "AuditType" => [
    [0] "FreshCredentials"
],

```

So it works for me.

---

<div class="post-metadata">

**Author:** ![riahc3](https://avatars.discourse-cdn.com/v4/letter/r/d6d6ee/32.png) [@riahc3](https://discuss.elastic.co/u/riahc3)\
**Post date:** [February 25, 2021, 2:13pm UTC](https://discuss.elastic.co/t/need-some-help-with-logstash-and-the-xml-filter-plugin/265227/4 "2021-02-25T14:13:54Z")

</div>

I get this error message:

```
[2021-02-25T13:49:41,101][ERROR][logstash.agent] Failed to execute action {:id=>:adfs, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Expected one of [\\t\\r\\n], \"#\", \"}\" at line 59, column 35 (byte 447) after filter {\n xml{\n store_xml => false\n source => [winlog]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/reload.rb:53:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:365:in `block in converge_state'"]}
[2021-02-25T13:49:44,195][ERROR][logstash.agent] Failed to execute action {:id=>:adfs, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Expected one of [\\t\\r\\n], \"#\", \"}\" at line 59, column 35 (byte 447) after filter {\n xml{\n store_xml => false\n source => [winlog]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/reload.rb:53:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:365:in `block in converge_state'"]}

```

But are you saying it works for you? Or it doesnt work as expected?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 25, 2021, 4:22pm UTC](https://discuss.elastic.co/t/need-some-help-with-logstash-and-the-xml-filter-plugin/265227/5 "2021-02-25T16:22:11Z")

</div>

> [@riahc3](#):
>
> `source => [winlog][event_data][param2]`

You need double quotes around "[winlog][event\_data][param2]"

---

<div class="post-metadata">

**Author:** ![riahc3](https://avatars.discourse-cdn.com/v4/letter/r/d6d6ee/32.png) [@riahc3](https://discuss.elastic.co/u/riahc3)\
**Post date:** [February 25, 2021, 4:40pm UTC](https://discuss.elastic.co/t/need-some-help-with-logstash-and-the-xml-filter-plugin/265227/6 "2021-02-25T16:40:49Z")

</div>

Thanks 🙂 Thats seems to have worked fine; Im gonna keep on breaking down the rest of the fields.

I do see that the field has been introduced as a unknown type: I would like to give it a type such as string, number, etc. How can I do this?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 25, 2021, 4:46pm UTC](https://discuss.elastic.co/t/need-some-help-with-logstash-and-the-xml-filter-plugin/265227/7 "2021-02-25T16:46:12Z")

</div>

The only way I can think of for you to see an unknown type is in kibana, if you have not done an index pattern refresh. Otherwise I think everything in logstash will be a string and in elasticsearch [dynamic field mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-mapping.html) will have typed things.

---

<div class="post-metadata">

**Author:** ![riahc3](https://avatars.discourse-cdn.com/v4/letter/r/d6d6ee/32.png) [@riahc3](https://discuss.elastic.co/u/riahc3)\
**Post date:** [February 25, 2021, 5:05pm UTC](https://discuss.elastic.co/t/need-some-help-with-logstash-and-the-xml-filter-plugin/265227/8 "2021-02-25T17:05:42Z")

</div>

Yup, thats problably it. I think I had to do that once.

Im having another issue (even though its dumb)

This works:

```
filter {
        xml {
                store_xml => false
                source => "[winlog][event_data][param2]"
                xpath => ["/AuditBase/AuditType/text()", "AuditType"]
				xpath => ["/AuditBase/AuditResult/text()", "AuditResult"]
				xpath => ["/AuditBase/FailureType/text()", "FailureType"]
				xpath => ["/AuditBase/ErrorCode/text()", "ErrorCode"]
        }
}

```

This does NOT work

```
filter {
        xml {
                store_xml => false
                source => "[winlog][event_data][param2]"
                xpath => ["/AuditBase/AuditType/text()", "AuditType"]
				xpath => ["/AuditBase/AuditResult/text()", "AuditResult"]
				xpath => ["/AuditBase/FailureType/text()", "FailureType"]
				xpath => ["/AuditBase/ErrorCode/text()", "ErrorCode"]
				xpath => ["/AuditBase/ContextComponents/Component/RelyingParty/text()", "RelyingParty"]
        }
}

```

As you can see that last field is inside another XML field; Maybe Im not getting it right. Could you clarify it for me?

Thank you

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 25, 2021, 5:54pm UTC](https://discuss.elastic.co/t/need-some-help-with-logstash-and-the-xml-filter-plugin/265227/9 "2021-02-25T17:54:06Z")

</div>

> [@riahc3](#):
>
> ```auto
> xpath => ["/AuditBase/AuditType/text()", "AuditType"]
> xpath => ["/AuditBase/AuditResult/text()", "AuditResult"]
> xpath => ["/AuditBase/FailureType/text()", "FailureType"]
> xpath => ["/AuditBase/ErrorCode/text()", "ErrorCode"]
> xpath => ["/AuditBase/ContextComponents/Component/RelyingParty/text()", "RelyingParty"]
> 
> ```

If you supply an option multiple times logstash will combine them. Usually it does that in exactly the way you would expect. From time to time I have found cases where it does not. So I only use a single instance of any given option

```
        xpath => {
            "/AuditBase/AuditType/text()" => "AuditType"
            "/AuditBase/AuditResult/text()" => "AuditResult"
            "/AuditBase/FailureType/text()" => "FailureType"
            "/AuditBase/ErrorCode/text()" => "ErrorCode"
            "/AuditBase/ContextComponents/Component/RelyingParty/text()" => "RelyingParty"
        }

```

That said, even with 5 options I get

```
 "FailureType" => [
    [0] "CredentialValidationError"
],
 "AuditResult" => [
    [0] "Failure"
],
"RelyingParty" => [
    [0] "N/A"
],
   "AuditType" => [
    [0] "FreshCredentials"
],
   "ErrorCode" => [
    [0] "N/A"
],

```

So I am not sure what you might be doing wrong.

---

<div class="post-metadata">

**Author:** ![riahc3](https://avatars.discourse-cdn.com/v4/letter/r/d6d6ee/32.png) [@riahc3](https://discuss.elastic.co/u/riahc3)\
**Post date:** [February 26, 2021, 8:45am UTC](https://discuss.elastic.co/t/need-some-help-with-logstash-and-the-xml-filter-plugin/265227/10 "2021-02-26T08:45:33Z")

</div>

Combining as one DOES seem to work..... (a bunch inside ONE xpath instead of multiple xpath)

---

<div class="post-metadata">

**Author:** ![riahc3](https://avatars.discourse-cdn.com/v4/letter/r/d6d6ee/32.png) [@riahc3](https://discuss.elastic.co/u/riahc3)\
**Post date:** [February 26, 2021, 8:47am UTC](https://discuss.elastic.co/t/need-some-help-with-logstash-and-the-xml-filter-plugin/265227/11 "2021-02-26T08:47:37Z")

</div>

Hmmm.........Ever since I implemented this though, Kibana is acting funky: Throwing me random 500 errors. And other errors:

{"statusCode":429,"error":"Too Many Requests","message":"[parent] Data too large, data for [\<http\_request\>] would be [1033145744/985.2mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1033145744/985.2mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=18361954/17.5mb, in\_flight\_requests=0/0b, model\_inference=0/0b, accounting=75981320/72.4mb]: circuit\_breaking\_exception"}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 26, 2021, 8:47am UTC](https://discuss.elastic.co/t/need-some-help-with-logstash-and-the-xml-filter-plugin/265227/12 "2021-03-26T08:47:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.