O365 Module and o365.audit.ModifiedProperties.*.*

I have the o365 module largely working great, however, one particular fields that needs further parsing is being problematic.
Filebeat is not finishing the parsing of all the values in this field, leaving a singular o365.audit.ModifiedProperties field that looks like json, like:

	"audit": {
		"ModifiedProperties": {
			"Action_Client_Name": {
				"NewValue": "DirectorySync"
			"MethodExecutionResult_": {
				"NewValue": "Microsoft.Online.DirectoryServices.DirectoryUniquenessException"
			"LastDirSyncTime": {
				"OldValue": "[\r\n \"2024-05-28T21:43:45Z\"\r\n]",
				"NewValue": "[\r\n \"2024-05-28T22:13:47Z\"\r\n]"
			"Included_Updated_Properties": {
				"NewValue": "LastDirSyncTime"
			"TargetId_UserType": {
				"NewValue": "Member"

I am trying to parse these values out to custom fields with logstash but nothing I have tried works.
I can start with a simple:

 json {
            source => ][o365][audit][ExtendedProperties][additionalDetails]"
            target => "[tl-custom][toParse][additionalDetails]"

But I get constants warnings of
Error parsing json {:source=>"[o365][audit][ModifiedProperties]", :raw=>{}
because apparently "{}" is seen, which is either null or empty or the literal {} brackets, I'm not entirely sure.
So I started fronting that json input with

 if ![o365][audit][ModifiedProperties] or !([o365][audit][ModifiedProperties] =~ ".+") {
            mutate {
                remove_field => ["[o365][audit][ModifiedProperties]"] 
                add_tag => ["TEST"]    

which seems to resolve that warning issue. However, the logs seem to disappear when I do that, I never see ModifiedProperties field after that. Before the change, I see ModifiedProperties; After the change, I do not and I do not understand why. I also never see any events with the TEST tag so I know the mutate to remove the field was not hit.
Has anyone parsed these fields before?

1 Like