XML filter not parsing

[tropas (Andy Mitchell)]

I'm using packetbeat to sniff network traffic (only http), which I'm then trying to parse (as it's all XML) with logstash.

my logstash file looks like:
filter {
if [method] == "PUT" {
xml {
source => "%{[http][request][body]}"
target => "xml_message"
add_tag => ["found-xml"]
}
}
}

output {
  if [method] == "PUT" {
    if [http][request][headers][content-type] == "application/xml" {
                stdout { codec => rubydebug }
                }
     }
         }

The output shows the body field does contain XML (i.e. packetbeat is capturing it), however logstash just refuses to parse the XML for me. also, it doesn't seem to add_tag either, but will remove fields if i put them in the same xml {} block.

The xml in the body of the payload is valid (and i've simplified it to just one tag value and it still doesn't parse.

Thanks

[wwalker (Walker)]

Why not use the beats input to take the traffic and let it handle the parsing on its own?

[tropas (Andy Mitchell)]

Apologies - I am using the beats input plugin (file not shown). The xml I’m trying to parse is in the http body...which gets presented by beats (in the ruby output) but trying to then parse this field with the xml filter just doesn’t seem to do anything

[wwalker (Walker)]

Do you have an example you can give...I don't understand why/where the beats plugin presents XML data.

[magnusbaeck (Magnus Bäck)]

Please show us an example document that wasn't processed correctly.

[tropas (Andy Mitchell)]

So, after extensive testing I've learnt a few things;

1. source => needs to map to a simple field - e.g. "xml_content" - to achieve this I used an earlier mutux to copy the field add_field => {"xml" => "%{[http][request][body]}" }

2. There appears to be a bug in packetbeat where if the http body > 1024 bytes it doesn't send it to either logstash or elasticsearch (tried both configurations).

I've got successful parsing otherwise. I'll raise a new bug for the payload size issue.