Xml-file too big for logstash?

Elastic Stack Logstash
21

that's what my filter is currently looking like by the way:

filter {
  xml {
        source => "message"
        target => "[theXML]"
        store_xml => true
        remove_namespaces => true
        force_array => false
        remove_field => ["message"]
      }

  ruby {
        path => '/home/christiane/splitData.rb'
        script_params => {field => "[theXML][Event]" target => "[theXML][Event]"}
       }
# ruby {
#       code => '
#               e = event.get("[theXML][Event][EventData][Data]")

#               if e.kind_of?(Array)
#                       e.each { |x|
#                               event.set(x["Name"], x["content"])
#                              }
#               else
#                       event.set(e["Name"], e["content"])
#               end
#               '
#       }
#  mutate {
#       copy => {
#               "[theXML][Event][System][Provider]" => "Provider"
#               "[theXML][Event][System][EventID]" => "EventID"
#               "[theXML][Event][System][Version]" => "Version"
#               "[theXML][Event][System][Level]" => "Level"
#               "[theXML][Event][System][Task]" => "Task"
#               "[theXML][Event][System][Opcode]" => "Opcode"
#               "[theXML][Event][System][Keywords]" => "Keywords"
#               "[theXML][Event][System][TimeCreated]" => "TimeCreated"
#               "[theXML][Event][System][TimeCreated][SystemTime]" => "SystemTime"
#               "[theXML][Event][System][EventRecordID]" => "EventRecordID"
#               "[theXML][Event][System][Correlation]" => "Correlation"
#               "[theXML][Event][System][Execution]" => "Execution"
#               "[theXML][Event][System][Channel]" => "Channel"
#               "[theXML][Event][System][Computer]" => "Computer"
#               "[theXML][Event][System][Security]" => "Security"
#               }
 #         }

  date {
        match => ["[theXML][Event][System][TimeCreated][SystemTime]", "YYYY-MM-dd HH:mm:s$
        timezone => "Europe/Berlin"
        }

  }
22

okay nevermind, it's not the other ruby part, it's the mutate filter. I guess it doesn't work like that (with copy) without the [@metadata]?

23

If I am reading it correctly, there is no such field. It should be [theXML][System][Provider]

24

I honestly don't know why the output looked like that, there's a field called [theXML][System][Provider]. I actually tried to comment it back in after trying around a few other things, and it works now, I really don't know why. The only thing left now is that there are still the "original" fields (called something like theXML.Event.System.Provider...) that I copied into the new fieldnames in the mutate-copy filter when I upload it in kibana. Is there a way to just delete them all without having to remove each one specifically?

25

If you use mutate+remove_field to remove theXML it will remove all the sub-fields too.

26

but I would have to use that after the first mutate filter and the date filter, right?

that's what it currently looks like:

mutate {
        copy => {
                "[theXML][Event][System][Provider]" => "Provider"
                "[theXML][Event][System][EventID]" => "EventID"
                "[theXML][Event][System][Version]" => "Version"
                "[theXML][Event][System][Level]" => "Level"
                "[theXML][Event][System][Task]" => "Task"
                "[theXML][Event][System][Opcode]" => "Opcode"
                "[theXML][Event][System][Keywords]" => "Keywords"
                "[theXML][Event][System][TimeCreated]" => "TimeCreated"
                "[theXML][Event][System][TimeCreated][SystemTime]" => "SystemTime"
                "[theXML][Event][System][EventRecordID]" => "EventRecordID"
                "[theXML][Event][System][Correlation]" => "Correlation"
                "[theXML][Event][System][Execution]" => "Execution"
                "[theXML][Event][System][Channel]" => "Channel"
                "[theXML][Event][System][Computer]" => "Computer"
                "[theXML][Event][System][Security]" => "Security"
                }
         }

  date {
        match => ["[theXML][Event][System][TimeCreated][SystemTime]", "YYYY-MM-dd HH:mm:ss.SSSSSS"]
        timezone => "Europe/Berlin"
        }

  mutate {
        remove_field => ["[theXML]"]
         }

but like that I only get one Event when I look at it in Kibana. Is there something wrong with it? I don't get an Error or anything, but just one Event.

27

Then I imagine the split filter did not find the field it was set to split.

28

But without the remove_field the split works perfectly now, is that possible if it didnt find the right field?

29

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.