Xml filter does not work

Elastic Stack Logstash
1

Hi,
Do you have any idea why this xml filter does not work? :slight_smile:

filter{
if [type] == "abc" {
dissect {
mapping => {
message => '<%{abc}>%{syslog_timestamp} %{+syslog_timestamp} %{+syslog_timestamp} %{logsource} %{rest}'
}
}
if[rest] =~ "123" {
xml {
source => "rest"
target => "rest"
xpath =>
["/1/2/3/333/text()", "333"
]
}
}
}
}

XML:
<1>
<2 ProductName="" ProductVersion="" ProductFamily="">
<3>
<333>19136</333>
</3>
</2>
</1>

3

It does not work because it is not valid XML. Element names must start with underscore or a letter..

input { generator { count => 1 message => '' } }
filter {
    mutate { add_field => { "rest" => '<a1> <a2 ProductName="" ProductVersion="" ProductFamily=""> <a3> <a333>19136</a333> </a3> </a2> </a1>' } }
    xml {
        source => "rest"
        target => "rest"
        xpath => { "/a1/a2/a3/a333/text()" => "333" }
    }
}

works just fine

       "333" => [
    [0] "19136"
]
4

Thank you, you are obviously right it was my mistake.

The exact fields below:

<?xml version="1.0" encoding="UTF-8"?>

After simple xml {} without xpath everty kibana shows right fields (with start rest.EventList and rest.MachineInfo) but after using xpath i do not get enything in newly created field, its empty

So, for some reason any of below does not work:
xpath => ["/Main/MachineInfo/MachineName/text()"]
xpath => ["/rest.MachineInfo/MachineName/text()"]
any idea?

5

That's not the right syntax. You can only have one xpath option, and it takes pairs of values, either as a hash or an array.

6

Hi,
No matter what I will choose, whether
xpath => ["/Main/MachineInfo/MachineName/text()", "MachineName"]
or
xpath => ["//MachineName/text()", "MachineName"]
the field is empty, any idea why?

7

Not without seeing the actual XML, no.

8 
<?xml version="1.0" encoding="UTF-8"?><Main><MachineInfo><MachineName></MachineName><AgentGUID></AgentGUID><IPAddress></IPAddress><OSName></OSName><UserName></UserName><TimeZoneBias></TimeZoneBias><RawMACAddress></RawMACAddress></MachineInfo><EventList ProductName="" ProductVersion="" ProductFamily=""><Event><EventID></EventID><Severity></Severity><GMTTime></GMTTime><OPGData></OPGData><UserInfo></UserInfo><ThreatName></ThreatName><PolicyName></PolicyName><TimeSZone></TimeSZone></Event></EventList></Main>

I hope that this will help, thank you in advance

9

Well, for that XML,

        xpath => [ "//MachineName/text()" , "MachineName" ]

produces

"MachineName" => [],

If you add some text to the MachineName element it will extract it

 "MachineName" => [
    [0] "Foo"
]

It is an array, always an array, so references to MachineName may not do what you expect.

10

I have tried that and nothing, field is empty. To be honest, no matter which field I will choose, all of them appear to be empty after xpath.
Is there some way to debug this?
Like I said earlier xml filter works fine, there is something wrong with xpath.

One more thing
XML splitted output into 3 main fields:
rest2.EventList and there rest
rest2.EventList.Event.GMTTime
and rest2.MachineInfo

From Kibana:
rest2.MachineInfo {
"TimeZoneBias": [
"-60"
],
"AgentGUID": [
"{dhkkjkh}"
],
"IPAddress": [
"x.x.x.x"
],
"RawMACAddress": [
"abc"
],
"OSName": [
"Windows 10 Workstation"
],
"UserName": [
"SYSTEM"
],
"MachineName": [
"xxx"
]
}

11

Clearly the example XML you posted is not the XML you are working on (no rest. anywhere). I understand you may be unable to post proprietary information but it is making impossible for me to see what the problem might be.

12

Current config:
filter{
if [type] == "abc" {
dissect {
mapping => {
message => '<%{abc}>%{syslog_timestamp} %{+syslog_timestamp} %{+syslog_timestamp} %{logsource} %{rest}'
}
}
if[rest] =~ "123" {
xml {
source => "rest"
target => "rest2"
xpath => {"//Main/rest2.MachineInfo/MachineName/text()" => "MachineName"}
}
}
}
}

xpath option is my variable, I have tried many, and field is still empty.

13

If your syslog timestamps look like "Feb 1 01:02:03" with two spaces between Feb and 1 then this will break tomorrow. You need to tell dissect that there are extra delimiters used for visual alignment. The -> after syslog_timestamp does that.

message => '<%{abc}>%{syslog_timestamp->} %{+syslog_timestamp} %{+syslog_timestamp} %{logsource} %{rest}'

You should not add rest2. to the xpath expression. I don't know what else to say. Given the XML format you show, if, and only if, there is a value in the MachineName element, then

        xpath => [
            "/Main/MachineInfo/MachineName/text()", "MachineName"
        ]

will extract it. I have to infer that the XML format you show is not actually what you are using.

14

Thank you for all your help, for some reason this xpath does not want to work. Grok works just fine, I will stick with it for this kind of logs.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.