XML filter not parsing

Elastic Stack Logstash
1

I'm using packetbeat to sniff network traffic (only http), which I'm then trying to parse (as it's all XML) with logstash.

my logstash file looks like:
filter {
if [method] == "PUT" {
xml {
source => "%{[http][request][body]}"
target => "xml_message"
add_tag => ["found-xml"]
}
}
}

output {
  if [method] == "PUT" {
    if [http][request][headers][content-type] == "application/xml" {
                stdout { codec => rubydebug }
                }
     }
         }

The output shows the body field does contain XML (i.e. packetbeat is capturing it), however logstash just refuses to parse the XML for me. also, it doesn't seem to add_tag either, but will remove fields if i put them in the same xml {} block.

The xml in the body of the payload is valid (and i've simplified it to just one tag value and it still doesn't parse.

Thanks

2

Why not use the beats input to take the traffic and let it handle the parsing on its own?

3

Apologies - I am using the beats input plugin (file not shown). The xml I’m trying to parse is in the http body...which gets presented by beats (in the ruby output) but trying to then parse this field with the xml filter just doesn’t seem to do anything

4

Do you have an example you can give...I don't understand why/where the beats plugin presents XML data.

5

Please show us an example document that wasn't processed correctly.

6

So, after extensive testing I've learnt a few things;

  1. source => needs to map to a simple field - e.g. "xml_content" - to achieve this I used an earlier mutux to copy the field add_field => {"xml" => "%{[http][request][body]}" }

  2. There appears to be a bug in packetbeat where if the http body > 1024 bytes it doesn't send it to either logstash or elasticsearch (tried both configurations).

I've got successful parsing otherwise. I'll raise a new bug for the payload size issue.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.