Automatic XML parsing, how to avoid array creation

Elastic Stack Logstash
1

Hello,

I want to parse Windows Event XML Log with logstash. I use MULTILINE input plugin and XML filter but i did not manage to have the output i want.

This is my input file:


<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Security-Auditing" Guid="{54849625-5478-4994-A5BA-3E3B0328C30D}"/>
    <EventID>5152</EventID>
    <Version>0</Version>
    <Level>0</Level>
    <Task>12809</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8010000000000000</Keywords>
    <TimeCreated SystemTime="2018-03-08T10:55:46.927024800Z"/>
    <EventRecordID>1315549</EventRecordID>
    <Correlation/>
    <Execution ProcessID="4" ThreadID="80"/>
    <Channel>Security</Channel>
    <Computer>srv-kms01.orangeobw.bw</Computer>
    <Security/>
  </System>
  <EventData>
    <Data Name="ProcessId">0</Data>
    <Data Name="Application">-</Data>
    <Data Name="Direction">%%14592</Data>
    <Data Name="SourceAddress">192.168.0.212</Data>
    <Data Name="SourcePort">65232</Data>
    <Data Name="DestAddress">255.255.255.255</Data>
    <Data Name="DestPort">1211</Data>
    <Data Name="Protocol">17</Data>
    <Data Name="FilterRTID">845803</Data>
    <Data Name="LayerName">%%14597</Data>
    <Data Name="LayerRTID">13</Data>
  </EventData>
</Event>

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
[...]

This my logstash config file:

input {

  file {
    path => "/Users/l3m0ntr33/Desktop/test.xml"
    sincedb_path => "/dev/null"
    start_position => "beginning"
    type => "WindowsEvent"

    codec => multiline
    {
      pattern => "</Event>"
      negate => true
      what => "next"
    }

  }
}

filter {
  if [type] == "WindowsEvent" {
    xml {
       store_xml => true
       target => "doc"
       source => "message"
       force_array => false
    }
  }
}

output {
  elasticsearch {
    hosts => ["127.0.0.1:9200"] 
    index => "event-test"
  }
 stdout { codec => rubydebug }
}

This the result from output console:

{
          "path" => "/Users/l3m0ntr33/Desktop/test.xml",
       "message" => "<Event xmlns=\"http://schemas.microsoft.com/win/2004/08/events/event\">\n  <System>\n    <Provider Name=\"Microsoft-Windows-Security-Auditing\" Guid=\"{54849625-5478-4994-A5BA-3E3B0328C30D}\"/>\n    <EventID>5152</EventID>\n    <Version>0</Version>\n    <Level>0</Level>\n    <Task>12809</Task>\n    <Opcode>0</Opcode>\n    <Keywords>0x8010000000000000</Keywords>\n    <TimeCreated SystemTime=\"2016-03-08T10:55:51.442649800Z\"/>\n    <EventRecordID>1315550</EventRecordID>\n    <Correlation/>\n    <Execution ProcessID=\"4\" ThreadID=\"92\"/>\n    <Channel>Security</Channel>\n    <Computer>srv01.local</Computer>\n    <Security/>\n  </System>\n  <EventData>\n    <Data Name=\"ProcessId\">0</Data>\n    <Data Name=\"Application\">-</Data>\n    <Data Name=\"Direction\">%%14592</Data>\n    <Data Name=\"SourceAddress\">0.0.0.0</Data>\n    <Data Name=\"SourcePort\">68</Data>\n    <Data Name=\"DestAddress\">255.255.255.255</Data>\n    <Data Name=\"DestPort\">67</Data>\n    <Data Name=\"Protocol\">17</Data>\n    <Data Name=\"FilterRTID\">845803</Data>\n    <Data Name=\"LayerName\">%%14597</Data>\n    <Data Name=\"LayerRTID\">13</Data>\n  </EventData>\n</Event>",
          "tags" => [
        [0] "multiline"
    ],
    "@timestamp" => 2018-03-19T14:11:00.041Z,
      "@version" => "1",
          "host" => "MBPdel3m0ntr33",
          "type" => "WindowsEvent",
           "doc" => {
           "System" => {
              "TimeCreated" => {
                "SystemTime" => "2016-03-08T10:55:51.442649800Z"
            },
                     "Task" => "12809",
                 "Provider" => {
                "Name" => "Microsoft-Windows-Security-Auditing",
                "Guid" => "{54849625-5478-4994-A5BA-3E3B0328C30D}"
            },
                 "Computer" => "srv01.local",
                  "Version" => "0",
                  "EventID" => "5152",
                "Execution" => {
                "ProcessID" => "4",
                 "ThreadID" => "92"
            },
                  "Channel" => "Security",
            "EventRecordID" => "1315550",
                 "Keywords" => "0x8010000000000000",
                    "Level" => "0",
                   "Opcode" => "0"
        },
        "EventData" => {
            "Data" => [
                [ 0] {
                       "Name" => "ProcessId",
                    "content" => "0"
                },
                [ 1] {
                       "Name" => "Application",
                    "content" => "-"
                },
                [ 2] {
                       "Name" => "Direction",
                    "content" => "%%14592"
                },
                [ 3] {
                       "Name" => "SourceAddress",
                    "content" => "0.0.0.0"
                },
                [ 4] {
                       "Name" => "SourcePort",
                    "content" => "68"
                },
                [ 5] {
                       "Name" => "DestAddress",
                    "content" => "255.255.255.255"
                },
                [ 6] {
                       "Name" => "DestPort",
                    "content" => "67"
                },
                [ 7] {
                       "Name" => "Protocol",
                    "content" => "17"
                },
                [ 8] {
                       "Name" => "FilterRTID",
                    "content" => "845803"
                },
                [ 9] {
                       "Name" => "LayerName",
                    "content" => "%%14597"
                },
                [10] {
                       "Name" => "LayerRTID",
                    "content" => "13"
                }
            ]
        },
            "xmlns" => "http://schemas.microsoft.com/win/2004/08/events/event"
    }
}

It's not so far from what i want but the problem is the array.
I don't want to have an array for the "" content.

What i would like is that every "<Data Name="DestAddress">255.255.255.255" create distinct field in the same event with the name "DestAddress" and the value "255.255.255.255".

I try to look at grok and other filter plugins but without success, any idea ?

Thanks a lot for your help

2

I don't want to have an array for the "" content.

The what?

What i would like is that every "255.255.255.255" create distinct field in the same event with the name "DestAddress" and the value "255.255.255.255".

You'll have to write some Ruby code in a ruby filter for that. Examples of that have been posted in the past.

3

I mean the XML parser create an object of type array for "EventData" XML block but i want this array to be split in different fields.

If i have:

  <EventData>
    <Data Name="SourceAddress">192.168.0.212</Data>
    <Data Name="SourcePort">65232</Data>
  </EventData>

With the above config i have this result:

        "EventData" => {
            "Data" => [
                [ 0] {
                       "Name" => "SourceAddress",
                    "content" => "192.168.0.212"
                },
                [ 1] {
                       "Name" => "SourcePort",
                    "content" => "65232"
                },

But i want to have two distinct fields in my event:

"SourceAddress" => "192.168.0.212",
"SourcePort" => "65232"

Any clue or example to develop such a ruby filter code ?

Thks

4

I'm not particularly familiar with this filter, and the docs are a bit vague, but it's possible that the force_content option may help.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.