XML Filter Help Required

Shaun_Wells · May 27, 2015, 12:32pm

Hello All, I'm trying to get my head around the XML Filter, I have the following configuration for my filter.

 xml {
        source => "message"
        target => "message_parsed"
        add_tag => ["xml_parsed"]

When I parse the following extract through logstash.

20150525T20:21:13 <stats><stats xmlns='jcs:stats:jsm'><current-online-user-count>1730</current-online-user-count><login-rate>0</login-rate><successful_logins>93645</successful_logins><failed_logins>84583</failed_logins><uptime>1900999</uptime></stats>
<statsxmlns='jcs:stats:delivery'><total-message-packets>5428196</total-message-packets><total-presence-packets>288328380</total-presence-packets><total-iq-packets>4977074</total-iq-packets><messages-in-last-time-slice>0</messages-in-last-time-slice><average-message-size>0</average-message-size></stats></stats>

I seem to end up with what looks like an ARRAY.. What I was hoping for was a JSON type document, where each specific element within the XML, is defined with it's own field. What I actually see on my console is the following - objects in an array, are not well supported in Kibana..

 "message_parsed" => {
        "stats" => [
            [0] {
                                    "xmlns" => "jcs:stats:jsm",
                "current-online-user-count" => [
                    [0] "10"
                ],
                               "login-rate" => [
                    [0] "0"
                ],
                        "successful_logins" => [
                    [0] "100"
                ],
                            "failed_logins" => [
                    [0] "5"
                ],
                                   "uptime" => [
                    [0] "1901060"
                ]
            },
            [1] {
                                      "xmlns" => "jcs:stats:delivery",
                      "total-message-packets" => [
                    [0] "1000"
                ],
                     "total-presence-packets" => [
                    [0] "100000"
                ],
                           "total-iq-packets" => [
                    [0] "100000"
                ],
                "messages-in-last-time-slice" => [
                    [0] "0"
                ],
                       "average-message-size" => [
                    [0] "0"
                ]
            }

I was expecting to see in my JSON document, something like

"user-count" : "value"
"login-rate" : "value"

Any ideas on how I can fix this, or am I misunderstanding what the XML filter should be doing here. Examples would be helpful..
Thanks.
Boardman

rafaltrojniak · May 27, 2015, 5:05pm

Hello,

The topic is interesting, I had bumped into it some time ago.
The reason why the elements are generated as arrays, is because nodes () can happen multiple times.
The problem above does not happen with attributes, as they have to be unique on the each node (according to XML standard). So if you xml would look like the problem would not happen.

I was trying to use 'xpath' option https://www.elastic.co/guide/en/logstash/current/plugins-filters-xml.html#plugins-filters-xml-xpath . According to documentation it should be able to solve your problems.
Unfortunately I was trying to achieve that, but I failed Don't know why neither those queries didn't work :
xpath => [
"failed_logins/text()", "x_failed_logins",
"failed_logins", "x_failed_logins2",
"/stats/stats", "x_stats",
"/stats/stats[0]", "x_stats0",
"/stats/stats[1]", "x_stats1",
"/stats/stats[0]/failed_logins", "x_failed_logins3"
]

I don't know what was wrong with above queries - they didn't throw any errors/warnings.

Regards,

rafaltrojniak · May 27, 2015, 5:38pm

Ah, found the problem.
There supposed to be a bug that is hit when xml has 'xmlns' attributes. https://github.com/logstash-plugins/logstash-filter-xml/issues/10
After removing them I was able to extract the values to the root of the event. They are still an arrays, but this should be possible to query.

Please see example rules here :

github.com

rafaltrojniak/logstash_rules/blob/xml_xpath/rules/xml_xpath.cfg

filter{
  if [format] == "xml_xpath" {
     xml {
          source => "message"
          target => "message_parsed"
          add_tag => ["xml_parsed"]
          remove_namespaces => true
          xpath => [
            "/stats/stats/failed_logins/text()", "x_failed_logins"
            ]
     }
  }
}

Documentation here :

github.com

rafaltrojniak/logstash_rules/blob/xml_xpath/doc.md

Formats
-------

### XML_Xpath example

#### Event metadaa

#### Fields :
* `message_parsed`
  * Description : XML contents
  * example values:
    * `{"stats"=>[{"current-online-user-count"=>["1730"], "login-rate"=>["0"], "successful_logins"=>["93645"], "failed_logins"=>["84583"], "uptime"=>["1900999"]}, {"total-message-packets"=>["5428196"], "total-presence-packets"=>["288328380"], "total-iq-packets"=>["4977074"], "messages-in-last-time-slice"=>["0"], "average-message-size"=>["0"]}]}`
    * `{"format"=>"xml_xpath", "message"=>"<stats><stats xmlns='jcs:stats:jsm'><current-online-user-count>1730</current-online-user-count><login-rate>0</login-rate><successful_logins>93645</successful_logins><failed_logins>84583</failed_logins><uptime>1900999</uptime></stats>\n<stats xmlns='jcs:stats:delivery'><total-message-packets>5428196</total-message-packets><total-presence-packets>288328380</total-presence-packets><total-iq-packets>4977074</total-iq-packets><messages-in-last-time-slice>0</messages-in-last-time-slice><average-message-size>0</average-message-size></stats></stats>", "@version"=>"1", "@timestamp"=>"2015-06-04T14:00:31.122Z", "host"=>"zip", "x_failed_logins"=>["84583"], "message_parsed"=>{"stats"=>[{"xmlns"=>"jcs:stats:jsm", "current-online-user-count"=>["1730"], "login-rate"=>["0"], "successful_logins"=>["93645"], "failed_logins"=>["84583"], "uptime"=>["1900999"]}, {"xmlns"=>"jcs:stats:delivery", "total-message-packets"=>["5428196"], "total-presence-packets"=>["288328380"], "total-iq-packets"=>["4977074"], "messages-in-last-time-slice"=>["0"], "average-message-size"=>["0"]}]}, "tags"=>["xml_parsed"]}`
* `x_failed_logins`
  * Description : Result of XPath query
  * example values:
    * `["84583"]`
    * `{"format"=>"xml_xpath", "message"=>"<stats><stats xmlns='jcs:stats:jsm'><current-online-user-count>1730</current-online-user-count><login-rate>0</login-rate><successful_logins>93645</successful_logins><failed_logins>84583</failed_logins><uptime>1900999</uptime></stats>\n<stats xmlns='jcs:stats:delivery'><total-message-packets>5428196</total-message-packets><total-presence-packets>288328380</total-presence-packets><total-iq-packets>4977074</total-iq-packets><messages-in-last-time-slice>0</messages-in-last-time-slice><average-message-size>0</average-message-size></stats></stats>", "@version"=>"1", "@timestamp"=>"2015-06-04T14:00:31.122Z", "host"=>"zip", "x_failed_logins"=>["84583"], "message_parsed"=>{"stats"=>[{"xmlns"=>"jcs:stats:jsm", "current-online-user-count"=>["1730"], "login-rate"=>["0"], "successful_logins"=>["93645"], "failed_logins"=>["84583"], "uptime"=>["1900999"]}, {"xmlns"=>"jcs:stats:delivery", "total-message-packets"=>["5428196"], "total-presence-packets"=>["288328380"], "total-iq-packets"=>["4977074"], "messages-in-last-time-slice"=>["0"], "average-message-size"=>["0"]}]}, "tags"=>["xml_parsed"]}`

#### Example sources

This file has been truncated. show original

Shaun_Wells · May 28, 2015, 10:00am

When you say that you've removed the 'xmlns' element, was this a manual thing you've done with my example.. As my daily log contains around 1400 entries a day, so how do I go about removing this.. Or is it the fact that I just specify an xpath for each element, as shown below.

"/stats/stats/failed_logins/text()", "x_failed_logins"
"/stats/stats/login_rates/text()", "x_login_rates"

etc...
etc..

rafaltrojniak · May 28, 2015, 7:09pm

Hey,

Yes, I removed them manually. I understand that this is impossible to be done on regular basis.

The problem is, that xpath handling looks to be buggy : https://github.com/logstash-plugins/logstash-filter-xml/issues/10 or I'm using it incorrectly.
When the xmlns attributes are present, I was not able to get any results from xpath queries.
I have no idea what is going one. Maybe it is failing, maybe the xpath processing is somehow affected by that xmlns. I really don't know.

Regards,

Shaun_Wells · June 4, 2015, 9:46am

I was thinking of doing the following and was not sure if this is possible, but could I parse the file through GROK and do a pattern match, stripe the offending field out of the log file and then pass the rest of the results to the XML Filter ?

Otherwise, I think my only other option is to do some sort of pattern match on all the separate elements, but I only want the value's not the tags.. So far I've managed to PATTERN match one element, but not sure how I just grab the value.

<current-user>1</current-user>
(?<currentuser><[\w-]+>[0-9]+<.[\w-]+>)

Give's me the following results:

{
  "currentuser": [
    [
      "<current-user>1</current-user>"
    ]
  ]
}

rafaltrojniak · June 4, 2015, 2:12pm

Hey,

Just got back to that issue with few days of experience more

The XPath is actually working fine - my bad. The namespaces just altered the names of the XML nodes which made my XPath rules not working properly. There are two ways to resolve that known to me :

Learn XPath with Namespaces support (still ahead of me)
Disable namespaces in xml plugin using remove_namespaces => true
And that econd one worked fine - see here :
Rules https://github.com/rafaltrojniak/logstash_rules/blob/xml_xpath/rules/xml_xpath.cfg
Doc https://github.com/rafaltrojniak/logstash_rules/blob/xml_xpath/doc.md
That generates additional field ( x_failed_logins ) which is result of query for /stats/stats/failed_logins/text()
Unfortunately that is still an array, as XPAth queries can return multiple results.

After further thinking : Why do you want to make this not an array?
You can always refer to the variable as [x_failed_logins][0] (0-element of an array )
If you insist on that, you can use mutate plugin to move variable from one place to the other.

About using grok to do that - well, it would work, but IMHO that will cause more problems than unpacking the event using XML and than using Xpath or moving field directly where you want it.

Regards,