"IndexAuditTrail failed to index audit event" warning in elasticsearch.log

Elastic Stack Elasticsearch
1

Hello,

I recently completed a rolling upgrade of elasticsearch to version 5.4.3 using the rolling upgrade instructions. After restarting the master node the following warning is flooding /var/log/elasticsearch/elasticsearch.log:

[2017-07-11T15:34:50,291][WARN ][o.e.x.s.a.i.IndexAuditTrail] [NODENAME] failed to index audit event: [access_granted]. internal queue is full, which may be caused by a high indexing rate or issue with the destination

  • It is worth noting, that this exact behavior occurred during an upgrade in 2 different beta environments when upgrading to different 5.x versions.
  • The cluster is new so the only indices being created are from .monitoring and .security-audit-log

cluster:

  • 3 master eligible nodes (they are also data, and ingest nodes)
  • 1 data only node
  • X-pack version 5.4.3
  • xpack.security.audit.outputs: [ index, logfile ] is set on all nodes in /etc/elasticsearch/elasticsearch.yml

Rolling restart steps:

  1. Disable shard allocation on data node
  2. Run synced flush
  3. Stop es
  4. upgrade
  5. wait for green
  6. reenable shard allocation
  7. Repeat on non masters then finally on master

Thanks in advance!

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.