Logstash with 'ecs-compatibility' unable to accept particular CEF Extension field

[wii (Wan Izzat Iskandar)]

Hi, I am still new with Elastic Stack. I have a problem with Logstash configuration which I have found no answer to for days. I have two Logstash configuration, 'ecs.conf' and 'cef.conf'. These two configs are created for me to play around getting the idea of how Logstash works and observe the ecs-compatibility plugin.

I have identified that this error was caused when the CEF extension field 'rt' was included in the log ingested. With 'ecs.conf', I received the error _cefparsefailure but with 'cef.conf', Logstash was able to read and dynamically mapping it fine.

ecs.conf (index pattern: ecs-compatibility):

cef.conf:

Here are the Logstash config file:

ecs.conf

cef.conf

Logstash log for ecs.conf pipeline for this problem:

My take is the ecs compatibility plugin have issues to be able to read the CEF extension field 'rt', which have something to do with the timestamp format of the field 'rt'.

[Badger]

The codec uses a timestamp normalizer to parse rt (deviceReceiptTime) into [@timestamp].

The normalizer parses a format which has optional fields ("MMM dd[ yyyy] HH:mm:ss[.SSSSSSSSS][.SSSSSS][.SSS][ zzz]", and uses parseBest to include as many fields as possible.

However, the rt field should always start with a 3 letter month abbreviation so when it hits the 2 at the start of 2022-04-26T... it blows up at index 0 of that rt string.

One option (and if you are new to logstash this may be unclear) would be to remove the cef codec, add a mutate filter to replace the rt keyword (including punctuation in front and behind it to avoid randomly editing words) with something that the cef filter will not know how to expand. Then connect two pipelines using a tcp output and a tcp input with a cef codec (not pipeline-to-pipeline communication, which ignores codes).

Please do not post images of text. They are not searchable and some people will not be able to view them. Just post the text and use the </> icon in the toolbar above the edit pane to preserve formatting when necessary. You will get better responses if you take the time to do this.

[wii (Wan Izzat Iskandar)]

I thank and appreciate you for the response.