Hello,
I've been trying out a logstash config file for a pretty large xml-file containing log events. Thanks to the help of someon on here it originally worked, but only for a smaller version of the file, with about 400 lines.
When I try it for the original file (1,6m lines) I always get an xmlparsefailure as well as split_type_failure (also rubyexception, but that's the next problem to worry about, unless it has something to do with it).
But when I just split the file in smaller sections of 100000 lines, they all individually work (except for the rubyexception).
Could the length of the file cause those errors or is it something else? And how can I fix it?
The problem is, the output I get in the shell is way too long, cant even see everything, and also this is the first time I'm working with anything logstash related, so I really don't know much. I tried to set the max_lines in the codec-part extra high, but still doesn't work.
Here's the current config file:
input {
file {
path => "/media/sf_Shared_Folder/test/logstash/zehn.xml"
start_position => beginning
mode => read
sincedb_path => "/dev/null"
codec => multiline
{
pattern => "^<\?xmldata .*>"
negate =>"true"
what => "previous"
auto_flush_interval => 1
max_lines => 20000000000
}
}
}
filter {
xml {
source => "message"
target => "[@metadata][theXML]"
store_xml =>true
remove_namespaces => true
force_array => false
remove_field => ["message"]
}
split {
field => "[@metadata][theXML][Event]"
}
ruby {
code => '
e = event.get("[@metadata][theXML][Event][EventData][Data]")
if e.kind_of?(Array)
e.each { |x|
event.set(x["Name"], x["content"])
}
else
event.set(e["Name"], e["content"])
end
'
}
mutate {
copy => {
"[@metadata][theXML][Event][System][Provider]" => "Provider"
"[@metadata][theXML][Event][System][EventID]" => "Event ID"
....
"[@metadata][theXML][Event][System][Channel]" => "Channel"
"[@metadata][theXML][Event][System][Computer]" => "Computer"
"[@metadata][theXML][Event][System][Security]" => "Security"
}
}
date {
match => ["[@metadata][theXML][Event][System][TimeCreated][SystemTime]", "YYYY-MM-dd HH:mm:ss.SSSSSS"]
timezone => "Europe/Berlin"
}
}
output {
stdout { codec => rubydebug{ metadata => true}}
}