Console XML works, but not file XML

Elastic Stack Logstash
5

The problem is your multiline codec. It is not combining lines into valid XML. The first messages are

<?xml version=\"1.0\" encoding=\"UTF-8\" ?>\n<Requests><Request id=\"34\">...<Failed>false</Failed>
</Request>\n<Request id=\"35\">...<Failed>false</Failed>

When using target the xml filter will actually parse the first one, but the xpath option is completely unforgiving of any junk around the XML.

Change your codec to use

pattern => "<Request "
negate => true
what => "previous"

Then you need some filters to clean up the results

if [message] =~ /<\?xml/ { drop {} }
mutate { gsub => [ "message", "<(/)?Requests>", "" ] }

That leaves you with valid XML. You have some typos in your xpath expressions you will need to clean.

I actually would not do it using xpath. I don't like that everything is an array. Instead I would use

target => "xml_value"
store_xml => true
force_array => false

Now I realize you want to everything to have different names, but to get the names you want you basically add an underscore before an uppercase letter (except at start of key name) then fold to lowercase. That is easy enough to do in ruby...

ruby {
    code => '
        x = event.get("xml_value")
        if x
            x.each { |k, v|
                newk = k.gsub(/(?!^)([A-Z])/, "_\\1")
                newk = newk.downcase
                event.remove("[xml_value][#{k}]")
                event.set("[xml_value][#{newk}]", v)
            }
        end
    '
}

If you are new to Ruby regexps, (?!^) is a negative lookahead assertion which means do not match when ^ is at the start of the pattern. So this pattern only matches inside a key value, not at its start.