Winlogbeats xml_query support for new line chars

[grants]

Running winlogbeat version 8.11.1 and trying to use a custom xml_query for the event logs. My issue is the data I need to match contains a new line sequence that is proving difficult to filter for.

Provided here in an include example of what I am trying to match, once I get the match working, I can apple this to an exclude filter I'm actually trying to make to ignore all the powershell noise.

---
winlogbeat.event_logs:
  - id: new-line-test
    fields:
      version: 0
    xml_query: |
      <QueryList>
        <Query Id="0" Path="Microsoft-Windows-PowerShell/Operational">
          <Select Path="Microsoft-Windows-PowerShell/Operational">
            *[System[EventID=4103]] and
            *[EventData[Data[@Name='Payload']='CommandInvocation(Out-Default): "Out-Default"&#13;&#10;']]
          </Select>
        </Query>
      </QueryList>
output.kafka:
  hosts: ["kafka:9092"]
  topic: winlogbeat

Notice the hex encoded chars for \r\n this filter works just fine in Event Viewer

These escaped new line chars do not work in PowerShell and if you use wevtutil this would be the corresponding structured query that would work:

<QueryList>
  <Query Id="0" Path="Microsoft-Windows-PowerShell/Operational">
    <Select Path="Microsoft-Windows-PowerShell/Operational">
      *[System[EventID=4103]] and
      *[EventData[Data[@Name='Payload']='CommandInvocation(Out-Default): "Out-Default"
']]
    </Select>
  </Query>
</QueryList>

I've tried escaped \r\n, I've tried literal new-line in my yaml file, and I've even tried \n hoping it would just work.

Any ideas how to get a structured query like this to work? Would this be considered a bug? There has to be something with the > or | scalar formatting and the xml nested that is causing a problem, but I can't get any of it to work.

I'd like to figure this out so I can use a single query for some of my logs and have the query itself filter out the noise instead of reading it all and dropping it in a processor. Main reason for this is the log volume I'm reading is pretty high and results in higher than expected resource utilization from Winlogbeat. I tend to read then drop nearly 90% of the logs.

[grants]

If needed, this is how you would test the powershell version:

wevtutil qe /sq:true /c:1 <path to xml file>

I'd expect any structured query that works for PowerShell should also work in the winlogbeat.yml config as a xml_query setting, but this doesn't seem to be the case with these issues over new lines.

[grants]

This specific xml_query setting will pass syntax checks and matches the powershell version, but will not actually read any events.

    xml_query: |
        <QueryList>
          <Query Id="0" Path="Microsoft-Windows-PowerShell/Operational">
            <Select Path="Microsoft-Windows-PowerShell/Operational">
              *[System[EventID=4103]] and
              *[EventData[Data[@Name='Payload']='CommandInvocation(Out-Default): "Out-Default"
        ']]
            </Select>
          </Query>
        </QueryList>

[andrewkroh]

Assuming the \r is required for the match then I think you were are on the right path by doing the escaping. But with YAML the escaped characters can only be used with the double-quoted string format.

From YAML Ain’t Markup Language (YAML™) Version 1.2

This is the only style capable of expressing arbitrary strings, by using “\” escape sequences. This comes at the cost of having to escape the “\” and “"” characters.

xml_query: "<QueryList>\n  <Query Id=\"0\" Path=\"Microsoft-Windows-PowerShell/Operational\">\n    <Select Path=\"Microsoft-Windows-PowerShell/Operational\">\n      *[System[EventID=4103]] and\n      *[EventData[Data[@Name='Payload']='CommandInvocation(Out-Default): \"Out-Default\"\n']]\r\n    </Select>\n  </Query>\n</QueryList>"

If you run these test cases (Go Playground - The Go Programming Language), you can see in the hex dump output that only the double-quoted version actually contains the 0x0D character.

[grants]

If I were to slam this all onto 1 line, the spacing between xml elements isn't needed and the double quotes inside most of this can be converted to single quotes.

This works in powershell:

<QueryList><Query Id='0' Path='Microsoft-Windows-PowerShell/Operational'><Select Path='Microsoft-Windows-PowerShell/Operational'>*[System[EventID=4103]] and *[EventData[Data[@Name='Payload']='CommandInvocation(Out-Default): "Out-Default"
']]</Select></Query></QueryList>

If I escape the " with \" into the yaml config I end up with this:

<QueryList><Query Id='0' Path='Microsoft-Windows-PowerShell/Operational'><Select Path='Microsoft-Windows-PowerShell/Operational'>*[System[EventID=4103]] and *[EventData[Data[@Name='Payload']='CommandInvocation(Out-Default): \"Out-Default\"
']]</Select></Query></QueryList>

And lastly replacing that newline with \n

<QueryList><Query Id='0' Path='Microsoft-Windows-PowerShell/Operational'><Select Path='Microsoft-Windows-PowerShell/Operational'>*[System[EventID=4103]] and *[EventData[Data[@Name='Payload']='CommandInvocation(Out-Default): \"Out-Default\"\n']]</Select></Query></QueryList>

This passes a test config, but when running it doesn't actually collect any events like it should.

Example winlogbeat.yml:

winlogbeat.event_logs:
  - id: new-line-test
    fields:
      version: 0
    #xml_query: "<QueryList><Query Id='0' Path='Microsoft-Windows-PowerShell/Operational'><Select Path='Microsoft-Windows-PowerShell/Operational'>*[System[EventID=4103]]</Select></Query></QueryList>"
    xml_query: "<QueryList><Query Id='0' Path='Microsoft-Windows-PowerShell/Operational'><Select Path='Microsoft-Windows-PowerShell/Operational'>*[System[EventID=4103]] and *[EventData[Data[@Name='Payload']='CommandInvocation(Out-Default): \"Out-Default\"\n']]</Select></Query></QueryList>"

The escaped xml_query was me testing to make sure the quotes were at least right around the xml and that the single quotes worked. The commented out version did collect while the other did not collect anything (it should have).

Maybe the better way is to support a xml_query reference to a file if new lines are going to be an issue like this. According to the xml docs, it seems I should be able to use 
 but when attempting to use this, I get a query invalid error from winlogbeat, I get the same error when using powershell, but this works with Event Viewer.

I've tried the sample you provided, but also the same result, passes syntax checks, but doesn't collect any events. Would this new line char need some additional escaping to pass through yaml and render with xml?

[grants]

One additional note, I've been using wevtutil to validate these queries with the structuredquery option which is what I thought this setting would support. If I use their standard filtering query option, the new line char never seems to match no matter what I do.

I'm not too sure how Winlogbeat queries for events and if its the same as wevtutil's structuredquery. I also haven't tried the experimental API flag.

[andrewkroh]

Sorry I haven't had time to investigate this further. Could you open a elastic/beats issue for this so that we can help find a way to pass those special characters through into Windows.