I recently completed a rolling upgrade of elasticsearch to version 5.4.3 using the rolling upgrade instructions. After restarting the master node the following warning is flooding /var/log/elasticsearch/elasticsearch.log:
[2017-07-11T15:34:50,291][WARN ][o.e.x.s.a.i.IndexAuditTrail] [NODENAME] failed to index audit event: [access_granted]. internal queue is full, which may be caused by a high indexing rate or issue with the destination
- It is worth noting, that this exact behavior occurred during an upgrade in 2 different beta environments when upgrading to different 5.x versions.
- The cluster is new so the only indices being created are from .monitoring and .security-audit-log
- 3 master eligible nodes (they are also data, and ingest nodes)
- 1 data only node
- X-pack version 5.4.3
- xpack.security.audit.outputs: [ index, logfile ] is set on all nodes in /etc/elasticsearch/elasticsearch.yml
Rolling restart steps:
- Disable shard allocation on data node
- Run synced flush
- Stop es
- wait for green
- reenable shard allocation
- Repeat on non masters then finally on master
Thanks in advance!