Recommendations for ingesting XML stream

Hey guys,

I'd like to ingest events that are continually streaming on a TCP port.

The stream format is XML, although it's not one event per line: the opening tag and closing tag are often on different lines. Here's a snip:


Any recommendations on convenient ways to receive this data? If it requires some transformation, what type of munging would you recommend?


Try using a multiline codec

 codec => multiline { pattern => '^</Current>' negate => true what => "next" auto_flush_interval => 1 }

That will combine lines that do not contain </Current> into the following line that does contain that.

Wow, thanks for the help! This is working very nicely!

Here's what I wound up with:

input {
  tcp {
    mode => "client"
    host => "my-host"
    port => "12345"
    codec => multiline {
      pattern => "<Operation1>|<Operation2>|<Operation3>|<Operation4>|<Operation5>"
      negate => "true"
      what => "previous"
    type => "resource-usage"
    tags => ["xml"]
filter {
  if "xml" in [tags] {
    xml {
      source => "message"
      target => "xml_content"

Followup question: I'd like my document to contain a field named "Operation" that indicates which XML element this stanza started with (e.g. "Operation1", "Operation2", etc as seen in the multiline pattern tag). Is there a handy way to do that in the XML filter?

Not that I know of, but you could dissect it

dissect { mapping => { "message" => "<%{Operation}>%{}" } }

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.