Upgrade Elasticsearch from 6.0.0 to 6.2.1 breaks XPack security audit log bulk indexing

[aaronloesattv (Aaron L)]

We are using the docker images of the elastic stack. we were upgrading from 6.0.0 to 6.2.1 and when we did, some of our nodes were logging a bunch of errors reporting that they were failing to bulk index the security audit logs. we had to downgrade to 6.1.3 in order for the errors to go away. below are the errors that were reported. Because we are using the docker images, i'm guessing that there is no need to upgrade XPack as the image comes with it. We are using the docker images provided by Elastic.co

[2018-02-09T21:06:17,276][INFO ][o.e.x.s.a.i.IndexAuditTrail] [hlvBERy] failed to bulk index audit events: [failure in bulk execution:
[0]: index [.security_audit_log-2018.02.09], type [doc], id [-2tjfGEBxtj2NAbo5pYd], message [StrictDynamicMappingException[mapping set to strict, dynamic introduction of [roles] within [doc] is not allowed]]
[1]: index [.security_audit_log-2018.02.09], type [doc], id [_GtjfGEBxtj2NAbo5pYd], message [StrictDynamicMappingException[mapping set to strict, dynamic introduction of [roles] within [doc] is not allowed]]
[2]: index [.security_audit_log-2018.02.09], type [doc], id [_WtjfGEBxtj2NAbo5pYd], message [StrictDynamicMappingException[mapping set to strict, dynamic introduction of [roles] within [doc] is not allowed]]
[6]: index [.security_audit_log-2018.02.09], type [doc], id [AWtjfGEBxtj2NAbo5pcd], message [StrictDynamicMappingException[mapping set to strict, dynamic introduction of [roles] within [doc] is not allowed]]
[7]: index [.security_audit_log-2018.02.09], type [doc], id [AmtjfGEBxtj2NAbo5pcd], message [StrictDynamicMappingException[mapping set to strict, dynamic introduction of [roles] within [doc] is not allowed]]
[8]: index [.security_audit_log-2018.02.09], type [doc], id [A2tjfGEBxtj2NAbo5pcd], message [StrictDynamicMappingException[mapping set to strict, dynamic introduction of [roles] within [doc] is not allowed]]
[9]: index [.security_audit_log-2018.02.09], type [doc], id [BGtjfGEBxtj2NAbo5pcd], message [StrictDynamicMappingException[mapping set to strict, dynamic introduction of [roles] within [doc] is not allowed]]
... and so on

[dadoonet (David Pilato)]

Moved to #x-pack

[Albert_Zaharovits]

Hello @aaronloesattv

Thank you for taking the time to report this!
It is indeed a bug. The audit index mapping has changed a little in 6.2.0 and the update does not handle it. We are working on.
In the meantime, you can manually update the mapping for the current audit index, here is an example request:

curl -u elastic:password -XPUT "localhost:9200/.security_audit_log-2018.02.11/_mapping/doc" -H 'Content-Type: application/json' -d'
{
  "properties": {
    "roles": {
      "type": "keyword"
    }
  }
}
'

This assumes the rollup for the index is daily and that today is 11th of Feb 2018 . This manual operation is only required once for the current audit index. Subsequent indices will be created correctly.

[aaronloesattv (Aaron L)]

Thank you for getting back to me @Albert_Zaharovits . Do you have a link to the issue that is tracking this that i can follow? We'll probably defer our upgrade to 6.2.1 until this is resolved.

[willemdh]

I'm experiencing the same issue on my monitoring cluster after updating to 6.2.1..

curl -u elastic -XPUT "https://esnode:9200/.security_audit_log-2018.02.19/_mapping/doc" -H 'Content-Type: application/json' -d'{ "properties": { "roles": { "type": "keyword" } }}'

solved the problem.