Xml-file too big for logstash?

Elastic Stack Logstash
21

that's what my filter is currently looking like by the way:

filter {
  xml {
        source => "message"
        target => "[theXML]"
        store_xml => true
        remove_namespaces => true
        force_array => false
        remove_field => ["message"]
      }

  ruby {
        path => '/home/christiane/splitData.rb'
        script_params => {field => "[theXML][Event]" target => "[theXML][Event]"}
       }
# ruby {
#       code => '
#               e = event.get("[theXML][Event][EventData][Data]")

#               if e.kind_of?(Array)
#                       e.each { |x|
#                               event.set(x["Name"], x["content"])
#                              }
#               else
#                       event.set(e["Name"], e["content"])
#               end
#               '
#       }
#  mutate {
#       copy => {
#               "[theXML][Event][System][Provider]" => "Provider"
#               "[theXML][Event][System][EventID]" => "EventID"
#               "[theXML][Event][System][Version]" => "Version"
#               "[theXML][Event][System][Level]" => "Level"
#               "[theXML][Event][System][Task]" => "Task"
#               "[theXML][Event][System][Opcode]" => "Opcode"
#               "[theXML][Event][System][Keywords]" => "Keywords"
#               "[theXML][Event][System][TimeCreated]" => "TimeCreated"
#               "[theXML][Event][System][TimeCreated][SystemTime]" => "SystemTime"
#               "[theXML][Event][System][EventRecordID]" => "EventRecordID"
#               "[theXML][Event][System][Correlation]" => "Correlation"
#               "[theXML][Event][System][Execution]" => "Execution"
#               "[theXML][Event][System][Channel]" => "Channel"
#               "[theXML][Event][System][Computer]" => "Computer"
#               "[theXML][Event][System][Security]" => "Security"
#               }
 #         }

  date {
        match => ["[theXML][Event][System][TimeCreated][SystemTime]", "YYYY-MM-dd HH:mm:s$
        timezone => "Europe/Berlin"
        }

  }
22

okay nevermind, it's not the other ruby part, it's the mutate filter. I guess it doesn't work like that (with copy) without the [@metadata]?

23

If I am reading it correctly, there is no such field. It should be [theXML][System][Provider]

24

I honestly don't know why the output looked like that, there's a field called [theXML][System][Provider]. I actually tried to comment it back in after trying around a few other things, and it works now, I really don't know why. The only thing left now is that there are still the "original" fields (called something like theXML.Event.System.Provider...) that I copied into the new fieldnames in the mutate-copy filter when I upload it in kibana. Is there a way to just delete them all without having to remove each one specifically?

25

If you use mutate+remove_field to remove theXML it will remove all the sub-fields too.

26

but I would have to use that after the first mutate filter and the date filter, right?

that's what it currently looks like:

mutate {
        copy => {
                "[theXML][Event][System][Provider]" => "Provider"
                "[theXML][Event][System][EventID]" => "EventID"
                "[theXML][Event][System][Version]" => "Version"
                "[theXML][Event][System][Level]" => "Level"
                "[theXML][Event][System][Task]" => "Task"
                "[theXML][Event][System][Opcode]" => "Opcode"
                "[theXML][Event][System][Keywords]" => "Keywords"
                "[theXML][Event][System][TimeCreated]" => "TimeCreated"
                "[theXML][Event][System][TimeCreated][SystemTime]" => "SystemTime"
                "[theXML][Event][System][EventRecordID]" => "EventRecordID"
                "[theXML][Event][System][Correlation]" => "Correlation"
                "[theXML][Event][System][Execution]" => "Execution"
                "[theXML][Event][System][Channel]" => "Channel"
                "[theXML][Event][System][Computer]" => "Computer"
                "[theXML][Event][System][Security]" => "Security"
                }
         }

  date {
        match => ["[theXML][Event][System][TimeCreated][SystemTime]", "YYYY-MM-dd HH:mm:ss.SSSSSS"]
        timezone => "Europe/Berlin"
        }

  mutate {
        remove_field => ["[theXML]"]
         }

but like that I only get one Event when I look at it in Kibana. Is there something wrong with it? I don't get an Error or anything, but just one Event.

27

Then I imagine the split filter did not find the field it was set to split.

28

But without the remove_field the split works perfectly now, is that possible if it didnt find the right field?