I'd like to apply a global tag to every message imported from the same log file, based on information found in the first five lines.
Background: I'm parsing XML files recorded on a remote system, but imported on a local source. Each XML file includes five lines at the beginning to declare xml version, stylesheet, and relevant information about the original source (name, ID, software version). After those five lines, the real messages begin. The main messages are being parsed perfectly, but I'd like to apply tags to them to reflect the source name, ID, and software version.
Ideally, each message would either have 3 additional fields for Laboratory, System ID, and Software Version, or 3 additional tags reflecting those values. I'm not concerned about the xml version or stylesheet info. At present, my config file will import the first five lines as separate messages, rather than as header info for the remainder of the log file. I've included the config file below for reference.
I've been bumbling about with "if" statements and grok filters with minimal success for hours now, so I'd appreciate any suggestions you might have.
Example XML log:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="automationErrorLog.xsl"?>
<!-- Laboratory: Dexters Lab -->
<!-- System ID: Brainalyzer 4000.1.26 -->
<!-- Software Version: v3.0 -->
<ErrorLog>
<Error timestamp="20180528090858" nodeID="4" nodetype="LAZR" errorcode="1234"
moreinfo="Brain size too small ">
<Error timestamp="20180528090858" nodeID="7" nodetype="TNT" errorcode="5678"
moreinfo="Deedee pushed self destruct">
</ErrorLog>
Logstash config:
input {
file {
path => "D:/Data/Kibana Imports/*/*/*.xml"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
if [path] =~ "ERROR" {
xml {
source => "message"
target => "Error"
xpath => ["/Error/@timestamp","ERRORtimestampString"]
}
mutate{
replace => { "ERRORtimestampString" => "%{ERRORtimestampString[0]}" }
remove_field => "host"
remove_field => "@version"
}
date {
match => ["ERRORtimestampString", "yyyyMMddHHmmss"]
target => "@timestamp"
}
mutate{
remove_field => "ERRORtimestampString"
}
output {
stdout {
code => rubydebug
}
}