Append metadata to every event in XML file being ingested to Elasticsearch

Elastic Stack Logstash
1

Hey, I have a xml file as follows and I want to append the metadata which are the first few lines of the document to every event in the xml file in subsequent lines.

The xml file is attached here. The expected output for the first event document is attached here.

2

I assume your xml file will have </events> and </eventdata>. If not it is not valid XML, but that can be fixed with a mutate+ gsub to append text to the message if needed.

Consume the entire XML as a single event using a file input and a multiline codec that never matches. For example

codec => multiline { pattern => "^Spalanzani" negate => true what => "previous" auto_flush_interval => 1 }

Then you can parse the XML and split on the event field

xml { source => "message" target => "theXML" store_xml => true }
split { field => "[theXML][events][0][event]" }

Then you just have mutate+rename all the fields to get them where you want them.

3

@Badger, Thats sounds good, but how do I capture metadata details like "deploymentid" and "siteid" and duplicate it to every event document?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.