CEF codec does not resolve extended fields

[jamn03]

I have an event flow that is processing Windows logs forwarded via syslog, parsing them, and sending them to ArcSight via the CEF codec. The list of extended fields to be included in the CEF message is variable depending on the event ID, and the list of CEF extended fields to be mapped is stored in the event field cef_fields

output {
    if [type] == "domain_controller" {
        udp {
            id => "arcsight_connector"
            host => "1.1.1.1"
            port => 514
            codec => cef {
                delimiter   => " "
                vendor      => "%{device_vendor}"
                product     => "%{device_product}"
                version     => "%{device_version}"
                severity    => "%{device_severity}"
                signature   => "%{device_signature}"
                name        => "%{os_event_description}"
                fields      => [cef_fields]
            }
        }
    }
}

The output message does not have the expected key/value pairs but instead is just a print of the cef_fields event field.

CEF:0|Microsoft|Microsoft Windows|1.0|Microsoft-Windows-Security-Auditing:4624|An account was successfully logged on|0|ceffields=["dvchost","deviceProcessName","cs5","cs5Label","cs1","cs1Label","cn1","cn1Label","shost","src","spt","duser","dntdom","duid","cs6","cs6Label","dproc","cs3","cs3Label","suser","sntdom","suid"]

A quick examination of the codec code makes me believe this should work.

values = @fields.map {|fieldname| get_value(fieldname, event)}.compact.join(" ")

Is there a better way to pass the cef_fields list?

[jamn03]

I didn't get an answer from the experts, but I did consult with others who have used the CEF codec before. The consensus was that passing an object as the fields parameter does not work. I suspect this is related to some weirdness with the event API not implementing an event field value which contains an array in a way that the actual array is passed to the CEF codec (which is what the codec code expects).

Thankfully, though, if you hard code a valid cef key in the fields array but the cef key isn’t present as a root event field then the codec just skips over it. So the solution is just to include all the keys (or all keys that may be used in your solution) in the fields array and then selectively map the fields you want to the event.

The valid CEF keys can be found in the codec github. Just search for ‘MAPPINGS’

logstash-plugins/logstash-codec-cef/blob/master/lib/logstash/codecs/cef.rb

Here’s what my output looks like now:

output {
    if [type] == "domain_controller" {
        udp {
            id => "arcsight_connector"
            host => "1.2.3.4"
            port => 514
            codec => cef {
                delimiter   => " "
                vendor      => "%{device_vendor}"
                product     => "%{device_product}"
                version     => "%{device_version}"
                severity    => "%{device_severity}"
                signature   => "%{device_signature}"
                name        => "%{os_event_description}"
                fields      => [
                	"dvchost"
                    "duser",
                    "dntdom",
                    "cs5",
                    "cs4",
                    "shost",
                    "cs6",
                    "cs3",
                    "cs1",
                    "src",
                    "duid",
                    "cn1",
                    "fname",
                    "sntdom",
                    "dvchost",
                    "deviceProcessName",
                    "suid",
                    "destinationServiceName",
                    "dpriv",
                    "spt",
                    "susr",
                    "cs2",
                    "dproc"
                ]
            }
        }
    }
}