Change log level for beats deployed by fleet managed elastic-agent

Elastic Stack Elasticsearch
1

I've set up Elasticsearch and Kibana on a couple of test VM's.
These will be running on Raspberry Pi's once I have it all figured out.
Hence, it's pretty important that I limit logging.

I am trying really hard to find all the different places where the various components are set to info level logging and reduce that to warning level.

The problem comes with Fleet deployed agents.
I notice that all of the beats that the agent is running are at info level logging.
Eg when I look at the output of
systemctl status elastic-agent
(which is really huge and I'm going to post the whole thing at the end)
has every one of the beats that it runs showing:
-E logging.level=info
Now, I've tried really really hard to find where this is configured in the Fleet settings, the Agent settings etc.

I did manage to find the place in each individual agents log settings, hidden right at the bottom of the page and really easy to miss, where you can set its log level, and these are all set to warning.

But this doesn't seem to affect the beats which the Fleet deployed agent starts.

I also tried adding both

agent.logging.level: warning
logging.level: warning

to the elastic-agent.yml but this also doesn't affect this particular logging.

Could some kind soul please end my torment by pointing me to the appropriate place to configure this logging down to warning level?

Thanks

Heres the output of systemctl that I mentioned where you will see that despite my best efforts for two days, I've been unable to change these:

● elastic-agent.service - Elastic Agent is a unified agent to observe, monitor and protect your system.
     Loaded: loaded (/etc/systemd/system/elastic-agent.service; enabled; preset: enabled)
     Active: active (running) since Sun 2023-07-16 16:37:42 NZST; 22min ago
   Main PID: 490 (elastic-agent)
      Tasks: 65 (limit: 4608)
     Memory: 665.7M
        CPU: 15.972s
     CGroup: /system.slice/elastic-agent.service
             ├─490 /opt/Elastic/Agent/elastic-agent
             ├─927 /opt/Elastic/Agent/data/elastic-agent-cdc5ba/components/metricbeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.enabled=true -E management.restart_on_output_change=true -E logging.level=info -E logging.to_stderr=true -E "gc_percent=\${METRICBEAT_GOGC:100}" -E metricbeat.config.modules.enabled=false -E http.enabled=true -E http.host=unix:///opt/Elastic/Agent/data/tmp/system-metrics-default.sock -E path.data=/opt/Elastic/Agent/data/elastic-agent-cdc5ba/run/system/metrics-default
             ├─928 /opt/Elastic/Agent/data/elastic-agent-cdc5ba/components/fleet-server --agent-mode -E logging.level=warning -E logging.to_stderr=true -E http.enabled=true -E http.host=unix:///opt/Elastic/Agent/data/tmp/fleet-server-default.sock -E path.data=/opt/Elastic/Agent/data/elastic-agent-cdc5ba/run/fleet-server-default
             ├─938 /opt/Elastic/Agent/data/elastic-agent-cdc5ba/components/metricbeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.enabled=true -E management.restart_on_output_change=true -E logging.level=info -E logging.to_stderr=true -E "gc_percent=\${METRICBEAT_GOGC:100}" -E metricbeat.config.modules.enabled=false -E http.enabled=true -E http.host=unix:///opt/Elastic/Agent/data/tmp/kibana-metrics-default.sock -E path.data=/opt/Elastic/Agent/data/elastic-agent-cdc5ba/run/kibana/metrics-default
             ├─947 /opt/Elastic/Agent/data/elastic-agent-cdc5ba/components/filebeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.enabled=true -E management.restart_on_output_change=true -E logging.level=info -E logging.to_stderr=true -E "gc_percent=\${FILEBEAT_GOGC:100}" -E filebeat.config.modules.enabled=false -E http.enabled=true -E http.host=unix:///opt/Elastic/Agent/data/tmp/log-default.sock -E path.data=/opt/Elastic/Agent/data/elastic-agent-cdc5ba/run/log-default
             ├─950 /opt/Elastic/Agent/data/elastic-agent-cdc5ba/components/filebeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.enabled=true -E management.restart_on_output_change=true -E logging.level=info -E logging.to_stderr=true -E "gc_percent=\${FILEBEAT_GOGC:100}" -E filebeat.config.modules.enabled=false -E http.enabled=true -E http.host=unix:///opt/Elastic/Agent/data/tmp/filestream-monitoring.sock -E path.data=/opt/Elastic/Agent/data/elastic-agent-cdc5ba/run/filestream-monitoring
             ├─960 /opt/Elastic/Agent/data/elastic-agent-cdc5ba/components/metricbeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.enabled=true -E management.restart_on_output_change=true -E logging.level=info -E logging.to_stderr=true -E "gc_percent=\${METRICBEAT_GOGC:100}" -E metricbeat.config.modules.enabled=false -E http.enabled=true -E http.host=unix:///opt/Elastic/Agent/data/tmp/beat-metrics-monitoring.sock -E path.data=/opt/Elastic/Agent/data/elastic-agent-cdc5ba/run/beat/metrics-monitoring
             └─966 /opt/Elastic/Agent/data/elastic-agent-cdc5ba/components/metricbeat -E setup.ilm.enabled=false -E setup.template.enabled=false -E management.enabled=true -E management.restart_on_output_change=true -E logging.level=info -E logging.to_stderr=true -E "gc_percent=\${METRICBEAT_GOGC:100}" -E metricbeat.config.modules.enabled=false -E http.enabled=true -E http.host=unix:///opt/Elastic/Agent/data/tmp/http-metrics-monitoring.sock -E path.data=/opt/Elastic/Agent/data/elastic-agent-cdc5ba/run/http/metrics-monitoring

Jul 16 16:37:42 kb-01 systemd[1]: Started elastic-agent.service - Elastic Agent is a unified agent to observe, monitor and protect your system..
2

Ugh I'm too old school. Looking on the command lines for how things are configured.
Next I'll be complaining about where are the init scripts and kids on my lawn.

So, although the command lines that the agent spawns to start the beats do show logging.level=info the actual configuration for the beats is dynamically pushed through the agent and can actually set the logging level however you want to, per agent, through the Fleet UI under the logging tab for each agent.
Its a bit of a pity this can't be configured as a default so that any new agents that get enrolled get their logging level set.
Instead one has to chase around new agents and set their log level manually.

Better than not being able to control it at all, though.

3

Hello @GibbsGreatly
you are correct, at the moment the only way to set logging level for fleet-managed agents is to set it on each agent enrolled in fleet.
This issue may be of interest to you since once that's implemented you will be able to set logging level via policy (only if you did not set a log level via fleet on the specific agent).

There's also a hack you could try (I don't think it is officially supported and may break at any moment): modify the logging level on elastic-agent.yml before installing (that is directly in the directory where you extracted the archive). I didn't try this recently but it worked at least till 8.8.0

Hope this helps :wink:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.