CPU at 100% with XPACK security Enabled - ES 7.9.0

[ilavajuthy (ila)]

Hi,
We are noticing high CPU after upgrading to 7.9.0 with XPACK security. Earlier we had 6.8.7 version without XPACK auth. We are using ECK deployment.
I see few posts on this issue, but none of the topic has resolution or suggestion so far.

hot_threads output shows lots of transport_worker thread working on RBAC
e.g

80.4% (402.1ms out of 500ms) cpu usage by thread 'Elasticsearch[elastica-logs-xpack-es-client-zd-2][transport_worker][T#5]'
4/10 snapshots sharing following 82 elements
java.base@14.0.1/java.util.HashMap.put(HashMap.java:613)
java.base@14.0.1/java.util.HashSet.add(HashSet.java:221)
org.Elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizedIndicesFromRole(RBACEngine.java:522)
org.Elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:352)

Is there a way to disable authorization?
Anyone faced this problem and have any recommendation?
We have around 2K indices and 13K aliases. We do have another bigger ES cluster still in 6.8.7 with load much higher than the current 7.9.0, but no performance issue noticed.
At the end all the ingestions are taking very long time. hot_threads across all the nodes are filled with only RBAC related activities by transport_worker thread.

Any pointers will be really helpful.

Thanks,
~ila.

[Christian_Dahlqvist]

You should upgrade to a more recent version as I believe version 7.9.0 has a memory leak. It sounds like you have a lot of aliases, but I will leave that for someone more knowledgable to comment on the potential impact of this.

[ilavajuthy (ila)]

Yes, we noticed the memory leak fix in 7.9.1.

   java.base@14.0.1/java.util.HashMap.put(HashMap.java:613)
   java.base@14.0.1/java.util.HashSet.add(HashSet.java:221)
   org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizedIndicesFromRole(RBACEngine.java:522)
   org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:352)

Does this code work on all the indices and aliases?

    static List<String> resolveAuthorizedIndicesFromRole(Role role, RequestInfo requestInfo, Map<String, IndexAbstraction> lookup) {
        Predicate<IndexAbstraction> predicate = role.allowedIndicesMatcher(requestInfo.getAction());

        // do not include data streams for actions that do not operate on data streams
        TransportRequest request = requestInfo.getRequest();
        boolean includeDataStreams = (request instanceof IndicesRequest) && ((IndicesRequest) request).includeDataStreams();

        Set<String> indicesAndAliases = new HashSet<>();
        // TODO: can this be done smarter? I think there are usually more indices/aliases in the cluster then indices defined a roles?
        for (Map.Entry<String, IndexAbstraction> entry : lookup.entrySet()) {
            IndexAbstraction indexAbstraction = entry.getValue();
            if (predicate.test(indexAbstraction)) {
                if (indexAbstraction.getType() != IndexAbstraction.Type.DATA_STREAM) {
                    indicesAndAliases.add(indexAbstraction.getName());
                } else if (includeDataStreams) {
                    // add data stream and its backing indices for any authorized data streams
                    indicesAndAliases.add(indexAbstraction.getName());
                    indicesAndAliases.addAll(indexAbstraction.getIndices().stream()
                        .map(i -> i.getIndex().getName()).collect(Collectors.toList()));
                }
            }
        }
        return Collections.unmodifiableList(new ArrayList<>(indicesAndAliases));
    }

Will reducing the alias help over here?