How to set `bulk_max_size` and `compression_level`?

[Craig_Rodrigues]

I have about 2000 Elastic agents (version 8.9.0) connected to a system with 3 Fleet servers (version 8.9.0).

We have about 20 different agent policies, because the various Elastic agents are sending
slightly different logs, and for certain cases we need to specify specific pipelines to process the logs.

At this link: Configure the Elasticsearch output | Fleet and Elastic Agent Guide [8.9] | Elastic

I see descriptions for bulk_max_size and compression_level.

How can I set those in the Elastic Agent policy in the Fleet UI?

I looked in the Fleet UI for editing an agent policy at:
kbn:/app/fleet/policies/{policy-id}/settings
and did not see a place where I could modify those fields

[stephenb (Stephen Brown)]

Hi @Craig_Rodrigues

It is edited at the output not the policy

Fleet -> Setting -> Output

In the yaml settings

[Craig_Rodrigues]

@stephenb Thanks for the response.
I have 2000 agents using 20 different agent policies.

In my use case, I only want certain agents to be affected by the bulk_max_size and compression_level settings.

I do not want these settings to apply to all 2000 agents.

Is that possible, or can I only specify these settings globally?

[stephenb (Stephen Brown)]

You can create different outputs one with settings one without and apply them as needed.

You will need to check of output is per policy or agent I don't recall AFK right now.

[leandrojmp (Leandro Pereira)]

@Craig_Rodrigues

I do not want these settings to apply to all 2000 agents.

Is that possible, or can I only specify these settings globally?

The output is per policy, so it will be applied on every agent under that specific policy.

[Craig_Rodrigues]

OK, so based on the advice @stephenb and @leandrojmp have given, I think I need to do this:

1. Go to Fleet -> Settings
   ( kbn:/app/fleet/settings )

2. Go to Outputs section. Right now, I only have one Output, labelled default:

3. Click on Add output to add another output

4. In the new output go to Advanced YAML configuration, add the settings for bulk_max_size and compression_level there:

5. Go to a particular agent policy
   Fleet -> Agent policies
   ( kbn:/app/fleet/policies/{policy id} )

6. Scroll down to Output for integrations

7. Change this output to point to the output I created in step 3.

Some of this is documented at Advanced YAML configuration

Does my understanding of the necessary steps seem right?

[Craig_Rodrigues]

@leandrojmp @stephenb Do the steps which I have described above seem right?

[leandrojmp (Leandro Pereira)]

Not sure what I could add, this is exactly what it is present in the documentation.

If you want this to just some agents you will need to first create the new output with this configuration, then create a new policy and use this output.

[Craig_Rodrigues]

Thanks for your help @leandrojmp and @stephenb