Ingesting O365 logs but having issues with specific mappings

[victor.nilsson]

Hi,

We are using Filebeat with the O365 module. These events get shipped to Redis which then Logstash fetches from. The logs are getting ingested but some of the events are having mapping issues with the field "o365.audit.Parameters".

We get these errors from logstash:

[2020-11-20T07:37:06,871][WARN ][logstash.outputs.elasticsearch][main][output_elasticsearch_redis] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"40f66e6a-7b49-425c-b362-6ace7be3bb63", :_index=>"office365-write", :routi
ng=>nil, :_type=>"_doc", :pipeline=>"office365-geoip"}, #<LogStash::Event:0x1e0feeab>], :response=>{"index"=>{"_index"=>"office365-2020.11-00001", "_type"=>"_doc", "_id"=>"40f66e6a-7b49-425c-b362-6ace7be3bb63", "status"=>400, "error"=>{"type"=>"mapper_pa
rsing_exception", "reason"=>"object mapping for [o365.audit.Parameters] tried to parse field [Parameters] as object, but found a concrete value"}}}}

We have the following mapping for the index which we got from exporting it from Filebeat:

GET _template/office365

{
  "office365" : {
    "order" : 100,
    "index_patterns" : [
      "office365-*"
    ],
    "settings" : {
      "index" : {
        "number_of_replicas" : "0",
        "mapping" : {
          "total_fields" : {
            "limit" : "10000"
          }
        }
      }
    },
    "mappings" : {
  "properties" : {
    "o365" : {
      "properties" : {
        "audit" : {
          "properties" : {
            "GroupName" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "ItemType" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "TargetUserOrGroupName" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "UserKey" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "ImplicitShare" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "AlertEntityId" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "EventData" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "Name" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "IntraSystemId" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "Item" : {
              "properties" : {
                "*" : {
                  "type" : "object",
                  "properties" : {
                    "*" : {
                      "type" : "object"
                    }
                  }
                }
              }
            },
            "OriginatingServer" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "Version" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "WebId" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "ClientAppId" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "MailboxOwnerUPN" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "SharePointMetaData" : {
              "properties" : {
                "*" : {
                  "type" : "object"
                }
              }
            },
            "CorrelationId" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "SessionId" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "MailboxOwnerMasterAccountSid" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "UniqueSharingId" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "Status" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "Parameters" : {
              "properties" : {
                "*" : {
                  "type" : "object",
                  "enabled" : false
                }
              }
            }
          }
        }
      }
    }

However it still complains about mapping errors.
What can we do to resolve this in the best way?

Thank you

[warkolm (Mark Walkom)]

What it's saying is that it expects an object in the o365.audit.Parameters field, as that is what is mapped. But it's only receiving a single item, which it can't then place into an object.

You'd need to look at the event in question a little closer, if you can, and figure out what it's missing.

[willemdh]

Hello @warkolm @victor.nilsson

We currently have a very similar issue with our o365 ingestion. (7.9.2)

Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"object mapping for [o365.audit.Parameters] tried to parse field [Parameters] as object, but found a concrete value"}

Indexation of all o365 logs is failing. This smells like a critical bug to me. Further investigation required..

Grtz

Willem

[willemdh]

Hello,

Not 100 % sure yet if it's a permanent fix, but for now I got ingestion working again by dropping the problematic field in filebeat.yml:

- drop_fields:
    when:
      equals:
        event.module: 'o365'
    fields: ["o365.audit.Parameters"]

Grtz

Willem

[victor.nilsson]

We're doing exactly the same. This has to be a bug with the default mapping installed by Filebeat

[willemdh]

Created https://github.com/elastic/beats/issues/22780, feel free to add more information.