How to set `bulk_max_size` and `compression_level`?

Elastic Stack Elastic Agent
,
1

I have about 2000 Elastic agents (version 8.9.0) connected to a system with 3 Fleet servers (version 8.9.0).

We have about 20 different agent policies, because the various Elastic agents are sending
slightly different logs, and for certain cases we need to specify specific pipelines to process the logs.

At this link: Configure the Elasticsearch output | Fleet and Elastic Agent Guide [8.9] | Elastic

I see descriptions for bulk_max_size and compression_level.

How can I set those in the Elastic Agent policy in the Fleet UI?

I looked in the Fleet UI for editing an agent policy at:
kbn:/app/fleet/policies/{policy-id}/settings
and did not see a place where I could modify those fields

2

Hi @Craig_Rodrigues

It is edited at the output not the policy

Fleet -> Setting -> Output

In the yaml settings

Screenshot 2023-08-22 at 9.11.44 AM
Screenshot 2023-08-22 at 9.11.44 AM1920×1122 103 KB

3

@stephenb Thanks for the response.
I have 2000 agents using 20 different agent policies.

In my use case, I only want certain agents to be affected by the bulk_max_size and compression_level settings.

I do not want these settings to apply to all 2000 agents.

Is that possible, or can I only specify these settings globally?

4

You can create different outputs one with settings one without and apply them as needed.

You will need to check of output is per policy or agent I don't recall AFK right now.

5

The output is per policy, so it will be applied on every agent under that specific policy.

1 Like
6

OK, so based on the advice @stephenb and @leandrojmp have given, I think I need to do this:

  1. Go to Fleet -> Settings
    ( kbn:/app/fleet/settings )

  2. Go to Outputs section. Right now, I only have one Output, labelled default:

image
image2468×430 33.4 KB

  1. Click on Add output to add another output

  2. In the new output go to Advanced YAML configuration, add the settings for bulk_max_size and compression_level there:

image
image860×330 3.51 KB

  1. Go to a particular agent policy
    Fleet -> Agent policies
    ( kbn:/app/fleet/policies/{policy id} )

  2. Scroll down to Output for integrations

image
image1784×358 35.1 KB

  1. Change this output to point to the output I created in step 3.

Some of this is documented at Advanced YAML configuration

Does my understanding of the necessary steps seem right?

1 Like
7

@leandrojmp @stephenb Do the steps which I have described above seem right?

8

Not sure what I could add, this is exactly what it is present in the documentation.

If you want this to just some agents you will need to first create the new output with this configuration, then create a new policy and use this output.

1 Like
9

Thanks for your help @leandrojmp and @stephenb

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.