Recommendations for ingesting XML stream

[dpr (David)]

Hey guys,

I'd like to ingest events that are continually streaming on a TCP port.

The stream format is XML, although it's not one event per line: the opening tag and closing tag are often on different lines. Here's a snip:

<Current>
  <Device>0x0011ee00001ee92a</Device>
  <TimeStamp>0x2509ad93</TimeStamp>
  <E1>0x00000000050e0ebb</E1>
  <M1>0x00000001</M1>
  <S1>Y</S1>
</Current>

Any recommendations on convenient ways to receive this data? If it requires some transformation, what type of munging would you recommend?

Thanks!

[Badger]

Try using a multiline codec

 codec => multiline { pattern => '^</Current>' negate => true what => "next" auto_flush_interval => 1 }

That will combine lines that do not contain </Current> into the following line that does contain that.

[dpr (David)]

Wow, thanks for the help! This is working very nicely!

Here's what I wound up with:

input {
  tcp {
    mode => "client"
    host => "my-host"
    port => "12345"
    codec => multiline {
      pattern => "<Operation1>|<Operation2>|<Operation3>|<Operation4>|<Operation5>"
      negate => "true"
      what => "previous"
    }
    type => "resource-usage"
    tags => ["xml"]
  }
}
filter {
  if "xml" in [tags] {
    xml {
      source => "message"
      target => "xml_content"
    }
  }
}

Followup question: I'd like my document to contain a field named "Operation" that indicates which XML element this stanza started with (e.g. "Operation1", "Operation2", etc as seen in the multiline pattern tag). Is there a handy way to do that in the XML filter?

[Badger]

@dpr

Is there a handy way to do that in the XML filter?

Not that I know of, but you could dissect it

dissect { mapping => { "message" => "<%{Operation}>%{}" } }