Parsing XML File -> xpath does not parse all fields in "event_data"

[jmohar]

Hello !
After hours of trying i can´t find "the light". I have the following customized Windows XML File... with data in [messages]...but xpath only recognize the first field (activity_id) ...and does not parse the "UserId" or "message"...simply ignoring these...
Can you point me to the right direction ? The XML File is customized in the "message" field.
I am new to logstash....

My XML:

 <?xml version="1.0" encoding="utf-8" standalone="yes"?>
<Events><Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='AD FS MFA'/><EventID Qualifiers='0'>2002</EventID><Level>4</Level><Task>0</Task><Keywords>0x80000000000000</Keywords><TimeCreated SystemTime='2020-03-10T07:05:31.383215200Z'/><EventRecordID>76971</EventRecordID><Channel>Application</Channel><Computer>FFM30FDSA401.asf.madm.net</Computer><Security/></System><EventData><Data>&lt;activity_id&gt;a3c26928-0c8f-4c66-fb01-0080020000fa&lt;/activity_id&gt;&lt;UserId&gt;joerg.mohar@metronom.com&lt;/UserId&gt;&lt;message&gt;joerg.mohar@metronom.com entered Token for Multi-Factor-Authentication&lt;/message&gt;</Data></EventData></Event></Events>

My logstash.conf:

input {
beats {
port => 5044
}}
filter {
if "ADFS_MFA" in [tags] {
xml {
#      source => "[event_data][param1]"
source => "message"
store_xml => false
force_array => false
xpath => ["activity_id/text()", "activity_id", "UserId/text()", "UserId", "message/text()", "MFA_Message"]
}}}

Do i need to mutate the "message" fields ? Thankful for any hints !

[Badger]

Please edit your post (use the pencil icon under the post) and format the text using markdown. If you select the body of the XML and click on </> then blockquote markup will be applied. Then do the same for the text of your configuration. You will see the appearance change in the preview pane to the right of the edit pane.

[jmohar]

Hello !
I formatted my text a little better now

During writing i figure out, that my XML-Message field seems to be no valid XML...so i trieg to "grok" this field with success:

if "ADFS_MFA" in [tags] {
grok {
match => { "message" => "<activity_id>%{GREEDYDATA:activity_id}</activity_id><UserId>%{GREEDYDATA:UserId}</UserId><message>%{GREEDYDATA:mfa_message}</message>"}
}

Now i have the fields "activity_id, UserID and MFA_Message" in Kibana...

Is there a better way to parse the (xml)message-field ?

[Badger]

I cannot speak to xpath, but you can parse it using a pair of xml filters.

    xml {
        source => "message"
        target => "[theXML]"
        store_xml => true
        force_array => false
        remove_field => [ "message" ]
    }
    mutate { gsub => [ "[theXML][Event][EventData][Data]", "^", "<a>", "[theXML][Event][EventData][Data]", "$", "</a>" ] }
    xml { source => "[theXML][Event][EventData][Data]"
        store_xml => true
        target => "foo"
        force_array => false
        remove_field => [ "theXML" ]
    }

will get you

       "foo" => {
         "UserId" => "joerg.mohar@metronom.com",
    "activity_id" => "a3c26928-0c8f-4c66-fb01-0080020000fa",
        "message" => "joerg.mohar@metronom.com entered Token for Multi-Factor-Authentication"
},

The mutate+gsub is required because your [EventData][Data] field looks like

<activity_id>a3c26928-0c8f-4c66-fb01-0080020000fa</activity_id><UserId>joerg.mohar@metronom.com</UserId><message>joerg.mohar@metronom.com entered Token for Multi-Factor-Authentication</message>

and you cannot parse something that has multiple elements at the root level using an xml filter, so I wrap the whole thing in <a> </a>