Performance impact of setting 'namespace' in Elastic agent policy config

Elastic Stack Elastic Agent
1

I have about 2000 Elastic agents (version 8.9.0) connected to a system with 3 Fleet servers (version 8.9.0).

We have about 20 different agent policies, because the various Elastic agents are sending
slightly different logs, and for certain cases we need to specify specific pipelines to process the logs.

In the Fleet UI for configuring an Elastic Agent policy,
the namespace is initially set to default .

On this page: Data streams | Fleet and Elastic Agent Guide [8.9] | Elastic

It mentions that the default naming scheme is:

<type>-<dataset>-<namespace>

If I have 20 different agent policies over 2000 elastic agents, do you recommend that from a performance perspective,
I configure 20 different namespaces, 1 unique namespace for each agent policy?

In the past, we left everything as namespace default, and this did not seem to be ideal from a performance perspective.

2

See related question here: How to specify ILM policies in Elastic agent policy config?

3

I did some research and consulted with Elastic and learned the following, which I believe
addresses my questions:

Specifying different namespaces in the agent policy
offers performance and organizational benefits.

  1. Data writes will be distributed across multiple namespaces
    instead of having all agents write to a single default
    namespace 'default'.
    This can reduce contention and potential bottlenecks of
    using a single namespace, which can lead to improved performance.

  2. Using separate namespaces allows for more fine-grained
    acess control and organization. This allows tailoring
    the permissions for specific policy requirements, which
    can enhance security and data governance.
    The trade-off of more namespaces is that each namespace
    introduces some overhead, such as resource consumption for
    management and administration.

Another approach to improving performance with thousands
of elastic agents is adjusting the number of shards within the index

  1. Increasing the shards in the index can improve performance by distributing the data
    more evenly across the index. Shard management introduces its
    own set of complexities such as increased resource usage and more
    complex indexing strategies.
2 Likes
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.