## Overview
The current state of [audit logging in Kibana](https://www.elasti…c.co/guide/en/kibana/current/xpack-security-audit-logging.html) is not sufficient for many users' needs. Kibana outputs only a few types of events, without much detail, in the same transport as regular log messages. This can be improved in many ways.
Enhancements in scope:
* More audit events and information regarding authentication -- e.g., log in and log out events
* More audit events for accessing objects
* Additional attributes for objects -- usernames, names, IPs, space IDs/names, object URLs, timestamps, authentication
* Additional information to differentiate specific user sessions
* Additional information to allow for correlation with Elasticsearch audit records
* Ability to include/exclude certain events and attributes
* Separate audit log transport with rotation capabilities
* Fail-safe to stop the Kibana process if audit records cannot be written
* Additional configuration to support all of the above enhancements
<details>
<summary>Current state vs. desired state...</summary>
------
### Current state
Audit records in Kibana are displayed in plaintext like so:
```
log [23:26:50.059] [info][audit][saved_objects_authorization_success][security] jdoe authorized to get config
log [23:26:50.067] [info][audit][saved_objects_authorization_success][security] jdoe authorized to find index-pattern
```
If JSON output is enabled:
```
{
"type": "log",
"@timestamp": "2020-02-18T14:58:44-05:00",
"tags": [
"info",
"audit",
"security",
"saved_objects_authorization_success"
],
"pid": 38933,
"username": "jojo",
"action": "get",
"types": [
"config"
],
"args": {
"type": "config",
"id": "8.0.0",
"options": {}
},
"eventType": "saved_objects_authorization_success",
"message": "jojo authorized to get config"
}
{
"type": "log",
"@timestamp": "2020-02-18T14:58:44-05:00",
"tags": [
"info",
"audit",
"security",
"saved_objects_authorization_success"
],
"pid": 38933,
"username": "jojo",
"action": "find",
"types": [
"index-pattern"
],
"args": {
"options": {
"perPage": 1,
"page": 1,
"type": [
"index-pattern"
],
"search": "*",
"defaultSearchOperator": "OR",
"searchFields": [
"title"
],
"fields": [
"title"
]
}
},
"eventType": "saved_objects_authorization_success",
"message": "jojo authorized to find index-pattern"
}
```
### Future state
Audit records should be written in a standard format ([ECS](https://www.elastic.co/guide/en/ecs/current/index.html)), should contain more information about the event that occurred and who originated the action, and fields should be configurable to include more or less information. Such an audit record would look something like this:
```
{
"@timestamp": "2019-12-05T00:00:02.000Z",
"event": {
"action": "get config",
"category": "saved_objects_authorization",
"duration": 453,
"end": "2019-12-05T00:00:02.453Z",
"module": "security",
"outcome": "success",
"start": "2019-12-05T00:00:02.000Z"
},
"host": {
"id": "5b2de169-2785-441b-ae8c-186a1936b17d",
"ip": "34.56.78.90",
"hostname": "hostname"
},
"http": {
"request": {
"body": {
"bytes": 887,
"content": "Hello world"
},
"bytes": 1437,
"method": "get",
"referrer": "https://blog.example.com/"
}
},
"labels": {
"spaceId": "default"
},
"source": {
"address": "12.34.56.78",
"ip": "12.34.56.78"
},
"url": {
"domain": "www.elastic.co",
"full": "https://www.elastic.co:443/search?q=elasticsearch",
"path": "/search",
"port": "443",
"query": "q=elasticsearch",
"scheme": "https"
},
"user": {
"email": "john.doe@company.com",
"full_name": "John Doe",
"hash": "D30A5F57532A603697CCBB51558FA02CCADD74A0C499FCF9D45B...",
"sid": "2FBAF28F6427B1832F2924E4C22C66E85FE96AFBDC3541C659B67...",
"name": "jdoe",
"roles": [ "kibana_user" ]
},
"trace": {
"id": "8a4f500d"
}
}
```
Note: in the example above, the `user.hash` (a hash of the `user.name` field) would not be included by default; it would be an optional field that could be included if the `user.name` needed to be excluded for privacy reasons.
------
</details>
## First Phase
Prerequisites (in progress):
- [x] Format audit records in JSON using the Elastic Common Schema (ECS) https://github.com/elastic/kibana/issues/52226
- [x] Modify Elasticsearch client to pass X-Opaque-Id header for unique events for correlation https://github.com/elastic/kibana/issues/62018
- [x] Collect audit logs for ES client https://github.com/elastic/kibana/issues/60119
- [x] Implement server-side sessions #17870
**Phase 1 implementation: #54836**
## Future Phase
- [x] Enriching events with session ID
- [x] Support for log rotation (prerequisite: #56291)
- [x] Additional attributes such as IP address (#127481) and user profile ID (#125932)
- [ ] Fail-safe to stop Kibana process if audit records cannot be written https://github.com/elastic/kibana/issues/60636
- [ ] Additional transport options (human-readable message formatting, multiple [appenders](../blob/master/src/core/server/logging/README.md))
- [ ] Support for including/excluding event attributes
- [ ] Include/exclude events _based on_ attributes (such as saved object type)
- [ ] Additional configuration to support the above